TestBash Brighton 2024

Let's build a language model together! Think it's impossible in the timeframe? Well, it's not going to be a general purpose Large Language Model. In fact, it's going to be a highly limited, incredibly Tiny Language Model. It's not about what the Language Model can do in this case, it's about what building it will show us.

I believe that to make truly effective use of a tool we need to have a rough understanding of how it works. But for many testers and automators LLMs seem to be a black box: we don't know how they work, so we're not sure when to use them or not to use them.

Making a TLM based on audience input is just one of the ways in which this talk aims to give a rough guide to how Generative AI works. So that we don't end up doing the equivalent of trying to push a nail into a wall with a screwdriver, or using a chainsaw instead of sandpaper.

What you’ll learn

AI as a buzzword is everywhere. It will steal our jobs, make us all obsolete and in the end: It will rule the world. We've been experiencing a shift in paradigms for two years and, most prominently, Large Language Models like LLaMA, ChatGPT or BARD are re-shaping industries and our everyday lives.

Using a Co-Pilot for Coding or Testing is seen as enhancing production and lowering barriers to entry.
But now that the uses of these LLMs are increasing rapidly:

• Who is testing them?
• And what actually is Quality in the age of AI?

In this talk, I want to provide results from my experience in projects of testing Large Language Models and regressive AI. I will explain the high-level function of a Large Language Model.

I will translate the components of a Copilot onto a newly thought testing pyramid from the component level to the system level. Now that we have a sort of framework to test LLMs, I will outline the metrics used and why testers will still be needed in the age of AI - maybe even more than ever.

What you’ll learn

In an era where test automation and AI are reshaping the software development landscape, my talk, "Enhancing Test Automation with Playwright and AI: A Journey of Innovation," offers a dive into my personal evolution within this dynamic space. As an Automation Engineer with over 15 years in the tech industry, I've witnessed first-hand the transformative power of cutting-edge technologies. My recent foray into using Playwright, coupled with AI, marked a pivotal shift in my approach to test automation, programming language transitions, and debugging methodologies.

This presentation will chart the course of my journey from initial experimentation with Playwright for Chrome extension testing to the strategic integration of AI tools like ChatGPT. I'll share how these technologies revolutionized my workflow, from effortlessly transitioning code snippets between JavaScript and TypeScript to generating robust test case scenarios and navigating complex debugging scenarios with unprecedented ease. The focus will not only be on the technicalities but also on the practical implementation and the tangible benefits observed in real-world projects.

Beyond personal insights, the session aims to equip participants with actionable strategies and innovative approaches to leverage Playwright and AI in their test automation endeavours. By dissecting challenges, solutions, and learnings, the talk will inspire attendees to explore new horizons in automation, enhancing their skill set and propelling their projects forward in the rapidly evolving digital landscape.

What you’ll learn

Have you ever been intimidated by any of the below - joining a new team or organization; working on a totally different Technology Stack; or working in an organization where testers are working in silos oblivious of the bigger picture impacting overall quality.

Well, the answer to all these can be in the practice of pair testing and doing it right when practising it with peers.

When I moved to a new continent, to join new teammates working in completely different Technologies - the introduction of pairing made my life easier. So, how did I do it?
Introduced the pairing practice to the culture of the Quality team.

This helped me to:

• understand Tech stack, work culture, product and processes
• grasp the length and breadth of the Tech stack by daring to pair with everyone: Engineering Manager, Product Owner, Developers and of course Testers in other teams

But, naturally, there was resistance initially.

In this talk, I share ways of introducing pair testing to teams and organizations. For example, start with scheduling a Pair testing session with peers at least once a week, which can be open for anyone to join in the organization. In my experience, it has been so impactful that it gets people curious to join initially and eventually makes them comfortable with the idea.
Everything sounds easier said than done until now; what if your peers have tight deadlines and developers are busy? In that case, let's turn to AI tools. These can be a game changer in pAIring and assisting. In my talk I will also discuss how these tools can be leveraged and speed up testing in multiple ways.

Join me in this talk as I share my experiences of pair testing with my peers as well as AI. And let's delve into the idea if the future of pair testing is only with AI.

What you’ll learn

As the use of AI in decision-making systems skyrockets, companies are under mounting pressure to ensure their AI deployments uphold ethical standards and inspire trust. Recent moves by many countries to introduce regulation targeting the use of AI is adding further pressure and the need for companies to show compliance.

Responsible AI is an emerging framework of principles and practices aimed at fostering the development of ethical, human-centered AI systems that prioritize robustness, reliability and trustworthiness.

This talk unpacks the essence of Responsible AI, shedding light on its core practices and highlighting the pivotal role testers can play building responsible and trustworthy AI systems.

While this is a great opportunity for testers it is not without its challenges and we will dive the obstacles and mindset shifts testers must navigate to collaborate in building Responsible AI systems.

We will end this talk with a recommended roadmap for testers who want to contribute to responsible AI systems.

What you’ll learn

Successful test automation implementations are notoriously difficult, and likely even the majority of teams are not getting the minimum necessary value from their efforts. Companies are increasingly turning to Artificial Intelligence (AI) as the answer, but so far the focus has been on making it easier to create the tests themselves. This can be useful, but it does not address the actual bottleneck: the requirement to maintain accurate test results over time. We need to focus on how AI can help testers rather than replace them.

In this talk, Titus will discuss the limitations of AI in its current form, and highlight data from multiple studies and surveys relating to how developers are actively using Large Language Models (LLMs) to identify their strengths and weaknesses. It is commonly known that LLMs hallucinate, so, similar to how testers are responsible for verifying the quality of the application they are testing, testers also need to verify the quality of the AI output in their workflows.

This talk will use the Selenium repository as an example to show how ChatGPT was used to automate the complicated Selenium build and release process. It will show how LLM tooling can provide value for both code generation and code management

The bottom line — to harness the full potential of AI in test automation, we must shift our focus from generating tests to empowering testers. By doing so, we can address the real problems facing test automation today, ensuring more sustainable and effective outcomes.
 

What you’ll learn

In the dynamic landscape of software testing, AI and automation have become increasingly prominent. However, this presentation delves into the critical issue of the industry's growing over-dependence on these technologies. It brings into focus the indispensable value of human testers, shedding light on their crucial role in deciphering complex systems and enhancing user experiences. Human testers' ability to pinpoint subtle, context-specific bugs and their comprehensive approach to ensuring software quality are often beyond the reach of AI.

Throughout the talk, we will unravel this theme by showcasing real-world case studies. These examples will vividly illustrate the limitations of AI in testing scenarios, particularly where intricate understanding and adaptability are required. The discussion will emphasize how AI, while beneficial in managing repetitive and straightforward tasks, lacks the nuanced judgment and creative problem-solving that human testers inherently possess.

By advocating for a balanced approach, the talk will argue for the integration of AI tools to augment efficiency but not at the expense of human expertise. The goal is to demonstrate that AI computational power is not going to replace human insights and a wise approach is needed to uphold the highest standards in software testing. This perspective aims to guide industry professionals in remembering that setting right tasks for AI is still human work.

What you’ll learn

More details about this Panel Discussion will be coming soon.

It's not a TestBash without 99 Second Talks!

The 99 Second Talks is the attendee's stage, an opportunity for you to come on stage and talk for, that's right, 99 seconds.

You can talk about anything, a testing topic you want to share, a personal experience, or an idea sparked by all the amazing talks, workshops, activities and conversations you've had for the past two days... the stage is yours, for 99 seconds!

This is also a great opportunity for you to kick-start your public speaking experience and/or give it a boost!

Our host will introduce you on stage and start the clock. As soon as the time's up, a noise will be heard and that's it: time's up!

What you’ll learn

Although I was extremely motivated while moving from a test lead to a quality coach role, I thought that my experience in testing and my skills as a facilitator and mentor were sufficient to embark on this role, but it was much more challenging than I had expected.

As a quality coach for many teams, I find myself dealing with agile-related problems, testing issues, and DevOps obstacles that were not visible to me when I was a member of one team. I failed for example to clarify and challenge management expectations while improving test coverage for the products under test. I learned that understanding the desired level of detail and making it clear to the team I’m supporting is essential before starting coaching. Besides that, I found it hard to master my new role and was it hard to provide visible results in a short period of time.

However, I learnt how to make team members aware of why we need these quality improvement actions and I helped them prioritize their backlog...

In a few months, I learned a lot in my new role. In this session, I want to share what I am doing as a quality coach, the struggles that I faced and what I will do differently next time.I will inform you about my strategies to avoid those difficulties before they occur. Besides that, I still see new challenges in my role coming up such as dealing with remote teams that have different backgrounds, coaching in the age of AI, etc.. all kinds of topics that could be a trigger for more discussion while leaving the room.

What you’ll learn

We all, often, think of ourselves as great people, unique, and maybe even the best in some sense (for better or worse). However, all these are usually until we need to interact with others.

How often do we get into a new team, or a new organisation and the happiness we usually feel at the start of the adventure start to dwindle over time?
There may be several reasons why this may happen. However, in a professional context, it can be due to the work culture, the leadership structure, the nature of work, and even our fellow colleagues. Yes, our colleagues, because it is always about them and not us, right?

The behavioural and cognitive attributes of professionals are usually not easy to control when recruiting for a position, or even when you are joining a new team. Yet, in today's dynamic professional landscape, achieving excellence and finding joy in our work is not only dependent on technical skills or qualifications; rather it hinges ever more on a deeper understanding of oneself.

In many years of working in various software development teams, I have experienced firsthand, the paramount importance of self-awareness in cultivating a productive and harmonious work environment. Even, some research has indicated that individuals with high levels of self-awareness are better equipped to navigate complex interpersonal dynamics, make informed decisions, and adapt to challenging circumstances. In the context of the workplace, self-awareness enables employees to recognise their unique talents and limitations, leading to improved collaboration, communication, and conflict resolution. Moreover, self-aware professionals are more adept at managing stress, fostering resilience, and maintaining a healthy work-life balance.

In light of the impact our actions and ways of being can have on us, and others, being self-aware becomes an important skill to cultivate as individuals. It is for this reason I say that "self-awareness is a moral responsibility at the workplace". Because, the more self-aware you are, the better colleague you will be to your team members.

The difficult question usually is: how do we cultivate self-awareness?
And, what can you do as someone in a position of leadership to foster a culture that values introspection and self-reflection?

I believe these are difficult, but important questions that need to be further discussed in the software development space. So, in this talk, I wish to share stories from my work-life experience to buttress the transformative power of self-awareness in shaping individuals into skilled professionals, great collaborators, effective leaders, and contributors to a healthy work environment.

What you’ll learn

We often talk about metrics and wonderful dashboards to showcase to stakeholders what’s going on and provide updates on progress.

Let me provide an alternative view on this. How about creating dashboards that help you as a team have conversations?

I fundamentally believe that metrics are powerful when they can help guide us. Metrics are about data.

• What is the data telling us?
• Do we need to look at different data here?
• Why are we even looking at this at all?

Through this power lens, we can start to think about the questions. What do we need as a team? What do our stakeholders want?

I have been experimenting with these thoughts. One example is tracking quality criteria (functional and non-functional bugs) to see where we are finding bugs, then digesting it to understand what we can learn from this.

I want to share these examples, so you can go away and think about what your own examples could be. Helping you and your team have conversations.

What you’ll learn

We all have our own unique testing journeys. Some parts are similar, while others are totally different. Unfortunately, not everyone’s journey is visible to the outside world.

In today's world, where everything is increasingly visible on digital platforms, a fundamental question emerges: "Is our testing journey visible too?" Surprisingly, only a negligible fraction of testers have an online portfolio today to showcase their skills, capabilities, and contributions. For both novice testers and seasoned professionals, this represents a significant opportunity to define their presence in the professional arena.

During this conference session, I aim to share my personal insights, experiences, and anecdotes acquired through the process of constructing my testing portfolio. I will talk about the art of crafting a compelling testing portfolio that not only distinguishes you but also substantiates your proficiency and trustworthiness.

What you’ll learn

For years now, we've been hearing a lot about the tester's mindset. While it remains as important as ever, I believe it has evolved. It has shifted from being the role of one person in the team who explores the application to find edge cases, thinking like a slightly deranged user, to a role that involves teaching this mindset to others. This includes developers, to help them write better unit and integration tests; product owners, to consider the unintended consequences of insufficiently bold ideas; and delivery leads, who may be hesitant to allocate time for proper test setup in the CI/CD pipeline.

I'll discuss how the tester role has evolved based on my experience and what I believe a modern tester needs to keep in mind to inspire their team to prioritise quality in all development activities. These days, critical thinking alone is no longer sufficient. Helping the team to move quickly while embedding quality into our processes has become more crucial than ever.

What you’ll learn

In the complex landscape of software testing, a people-first test manager stands out as a catalyst for positive team dynamics. More than overseeing test procedures, this role is about recognizing and valuing the people within the testing team. An inclusive Test Manager strives to create an environment where team members feel seen, heard, and appreciated. By fostering an inclusive culture that values each team member's unique strengths, a supportive Test Manager not only ensures the effectiveness of testing processes but also contributes to the overall well-being and job satisfaction of the team.

In this presentation, we're going to explore how Test Leaders are not just ensuring the functionality of software but are shaping the culture and success of the entire organization.

What you’ll learn

The newest development in technology always affects testing: first, it was the drive to “replace” testing with automation, and now with Artificial Intelligence. Why is the human contribution to testing consistently doubted and misunderstood? As the Ministry of Testing describes, testing is unpredictable and creative in nature. This makes it a uniquely human task, which can leverage but not be replaced by technology.

This talk will celebrate the creative problem-solving that is software testing. I will reflect on my academic and professional experiences in fine art and software testing, highlighting how you can apply the guidance given to creatives to your software testing problems. We’ll discuss how to think out of the box through the “Candle Problem”, a classic creative problem-solving experiment designed to highlight the cognitive bias of functional fixedness.

I will first highlight the principles shared by testing and traditionally artistic pursuits. This includes verifying that a piece (of software or art) offers its intended value to different stakeholders, and iteratively finding and solving its flaws. I will also discuss how testing and art both explore a product or idea in all its nooks and crannies and how this exploration samples from an infinite range of parameters and values. Art is never finished, only paused; like in testing, you must decide when a piece has achieved its goals.

I will then offer 4 problem-solving practices taught to art students that I’ve found invaluable in my day-to-day work in QA:

1. Turn it upside down. Does the problem appear differently from another perspective? What can you see that you couldn’t before?
2. Look at the bigger picture. Work on a problem, then step back. If our goal is a positive user experience, we’ll need to take a step back from tests and sprints to look at the application’s broader quality goals.
3. Practice telling a story. At art school, we would do workshops where we had 5 minutes to create an impromptu story about an item. The stories did not reflect our technical skills, but rather our ability to visualize and use imagination. My favourite part was seeing the different stories that each person would tell from the same item. This is why it’s important for testers to collaborate within and outside their teams when testing a final product.
4. Structure up. If the foundation of your drawing isn’t structurally sound, the end result will be poor, no matter how many details you add. The same goes for software applications; it doesn’t matter what the latest and greatest feature is if the entire application doesn’t work well.

What you’ll learn

Testing can be a powerful tool to discover information to empower stakeholders to make better, more informed decisions. But what if we need information about our mental health and we’re the stakeholders of our bodies and minds?
My mental health has been stable for so long that I started wondering if I still needed my medication and if it was safe to quit it. I thought that, by experimenting, I’d find answers to my questions. Can a healthy lifestyle and the tools learned through therapy be enough for me to live a happy life?

Designing an experiment

My approach was semi-scientific. I had a lot of questions (my hypothesis) and the purpose of my experiment was to gather information. I made assumptions regarding the outcome and I made a plan: I’ll start training to do the “Camino de Santiago” (a pilgrimage) and by working out, my body will get the happy hormones it needs to survive. Little did I know things weren’t going to go according to plan.

The tools that helped me

Some testers keep testing notes to not rely on our memory and for more effective debriefing, so I used my bullet journal to keep track of my mood, feelings, and my workout routine. I don’t know if I’d be here today if it wasn’t for my mood tracker, which became a solid piece of evidence.

What you’ll learn

It's not a TestBash without 99 Second Talks!

The 99 Second Talks is the attendee's stage, an opportunity for you to come on stage and talk for, that's right, 99 seconds.

You can talk about anything, a testing topic you want to share, a personal experience, or an idea sparked by all the amazing talks, workshops, activities and conversations you've had for the past two days... the stage is yours, for 99 seconds!

This is also a great opportunity for you to kick-start your public speaking experience and/or give it a boost!

Our host will introduce you on stage and start the clock. As soon as the time's up, a noise will be heard and that's it: time's up!

What you’ll learn