Agile for AI and Data Science

The challenges brought by digitalization and a fast-pacing market call for an agile approach to product/software development and organization structure.

In our digital world, being able to respond to change and quickly adapt are a must. Decision-making is evermore data-driven, and technologies such as Artificial Intelligence and Machine Learning are being employed in a constantly increasing amount of projects and application sectors.

Traditional Project Management and product development approaches are being abandoned in favour of agile practices and frameworks, with the scope of improving time to market, increasing productivity, delivered value, and outcome quality, while reducing costs and risks by early detection.

Agile Manifesto

But what does being agile truly mean? It is a cultural shift and a change of mindset. Practitioners need to step into novel roles and learn new ways of interacting with each other. Developing teams should be cross-functional and stop working in silos. As they are responsible for the technical quality of the product, they get more committed and should self-organize.

Agile Team

As a manager of an agile project, it is important to be a leader instead of a boss. Practically this means empower the team, inspire people and being able to coach and influence them with charisma, vision and innovation capacity. Being able to create a culture of trust enables the team to discuss issues — and therefore solving them —, propose new ideas, learn and share knowledge.

Agile Leader

As an organization working with agile, it is essential to have alignment and coordination across all areas, build a culture of transparency and trust, break silos and become more value-driven and user-oriented.

AI and Data Science projects are no different: they may also benefit from the application of agile methodologies and mindset, while at the same time posing specific challenges. Such projects may be more difficult to estimate, require a longer time investment before return (in contrast with the quick delivery cycles characteristic of traditionally agile-lead projects), and demand that crucial technical tasks to be prioritized in competition with more immediate business-valued development. On the other hand, fail quickly, evaluate and learn from experience, adjust and improve are typical agile trademarks which would positively impact an AI or Data Science project. Best coding techniques, continuous integration and deployment and DevOps best practices are essential for the success of such projects and walk hand in hand with an agile development approach.

Miraier Francesca covered these topics in her half-day lecture “Introduction to Agile for Data Science” on February 26th, part of the CAS AI Management at HWZ. The class started off with Understanding Agile, and specifically how an Agile Team and Leadership can look like and what their key characteristics are. Of course an agile approach impacts how a project is managed and how a product is developed, for this reason topics like incremental delivery, constant development speed and limited Work in Progress (WIP) were amply discussed. A quick overview of the most common Agile Frameworks at both Team and Organization levels gave the participants enough tools, and hopefully enough curiosity, to dig more into the various practices, roles and artifacts. Last but not least, the class put into focus the agile principles and tools fine tuned for a data science project.

However, we believe that just learning about it is not enough for embracing an agile mindset. Therefore, each topic in the lecture was associated with a practical activity, a hands-on exercise of what we know to be a best practice while working in an agile set-up. Our experience as agile trainers has thought us that when people live the agile implementation on their own skin and see the benefits with their own eyes, not only they end up being on board with the new culture but they are fully committed to integrating it in their daily work!

Obviously a virtual set-up poses a non-trivial set of challenges for hands-on activities. Non-verbal interactions are somewhat more limited via a webcam, and effective communication may be quite complex if the people in the work group do not know each other well. Fortunately after one year of pandemic and remote working we are all used to interact in a virtual context. From a teacher’s perspective, supervising multiple virtual groups is not an easy task: in the course the participants were divided into breakout rooms and had to perform different assignments within short iterative cycles, but enforcing the time-boxing of each step turned out to be tricky but still possible. Moreover, the collaborative virtual tool limitations are a constraint for the working group creativity and out-of-the-box thinking, particularly affecting exercises of self-organization and critical thinking. Nonetheless, using Goolgle Jamboards it was possible to get a taste of what working agile really means.

self organizing team

Learn more about the lessons learned of teaching Agile virtually in Francesca’s LinkedIn article.

Francesca was also present in the CAS AI Management at HWZ in 2020 with a one-hour guest lecture, you can read all about in this post on the HWZ blog (in German).

Do you want to know more about being agile, how to implement agility in your set-up, coach your team or get fast-tracked on agile best practices? Our Certified Scrum Masters will be happy to support you, do not hesitate to contact us!