Timeline of an Agile Data Environment: A Detailed View (Part 3 of 3)
In the last part of our series, we examine the detailed activities of an analytics team before, during, and after a sprint. You can compare your team’s work flow to the activities list in this article to see opportunities to become even more agile.
In Part 1 of this series, we explored the benefits of using agile techniques, and some of the cultural changes, processes, and tools needed to make it work. In Part 2 we looked at the big picture of how to adapt agile practices to your team. Now let’s take a detailed look at how this would affect the day-to-day activities.
In the following timeline and discussion, we will assume that a project is going to last at least several months, involve four or more data engineers and analysts, and have a full-time dedicated product owner (see Part 2 for a discussion on how to adapt these processes to organizations of different sizes). For a project of that size, the flow of tasks could follow a timeline similar to this:
Preparing for a Sprint
Product management should always be working at least two weeks ahead of development teams. The product manager is actively gathering requirements and encapsulating them in user stories, which are then placed in the backlog in order of priority. Stories that need technical input and exploration should be assigned to a developer as an investigation story/task in the sprint ahead of when they are expected to be implemented. If business requirements are fuzzy or the implementation isn’t clear, a prototype story can be created in a previous sprint to allow another iteration. Business sponsors need to be actively involved in defining and prioritizing the stories.
Data and analytics projects present some unique challenges for building stories and sprints. In every project there are data sources, some of which provide easy access to data while others may be blocked by legal, technical, or organizational obstacles. Master data management requires a big picture view of how all of the data fits together and is difficult to start quickly and break into small pieces, and some projects get buried under a mountain of legacy reports that all need to be analyzed and replaced.
These factors tend to force teams into thinking they need to do months of analysis and fall into a waterfall approach, but it is critical to find small pieces that can be completed in one sprint, get started as soon as possible, and show tangible results in each sprint. Although the product manager and data architect continue to perform analysis and design as part of each sprint, don’t wait for a final design or plan. Get your developers started right away on the things they can do to show value and keep business stakeholders engaged.
For data warehouse projects, a bus matrix can be used to show how stories fit together. In general, each dimension and fact table should have its own story. Data modeling should be done as part of design stories at least one sprint ahead of their implementation. Data sourcing from files, databases, and APIs should be done a sprint ahead of data-model-building tasks. Dimensions should be listed as predecessors of fact tables. Notify DBAs or other admins well in advance if you will need resources or permissions.
Originally published at https://tdwi.org
