STARWEST 2021 Concurrent Session : Deciphering the Way Data is Tested


Thursday, October 7, 2021 - 1:30pm to 2:30pm

Deciphering the Way Data is Tested

Accurate data interpretation and forecast can turn out to be the game changers in business decisions. The correct set of data is indispensable not only for decision-making but for technical solutions using AI/ML, deep learning, etc. Our aim is to provide a solution to challenges faced while data validation in:

  • Big data and AI/ML
  • Heterogeneous data format
  • Inadequacy of data anomaly identification
  • Huge data sets and a real-time stream of data
  • Understanding the data sentiment
  • Continuous testing and monitoring

The research employed an open-source solution for the implementation. Apache Kafka was used to gathering Batch data and Streaming data (Sensor/Logs). Apache Spark Streaming consumed the data from Kafka in Realtime and carried out the validations in Spark Engine. Further in the workflow, the data was stored in Apache Cassandra and then configured in Elasticsearch and Logstash to generate real-time reports/graphs in Kibana. The proposed tool is generic as well as highly configurable to incorporate any open-source tool you require to work within streaming, processing, or storing the data. The system includes configuration files where every single detail of the dependent tool used is appended and can be modified according to the needs.


Rajni Singh has worked in nearly every software development role - dev, test, DevOps, security, performance, and program management in her career. Integrating, delivering, and managing AI-based testing and Testing an AI application. A passionate QA, blogger, trainer, speaker, Rajni thrives excellence in software test quality. Focussing on new testing avenues in emerging technologies like big data, AI/ML, conversational AI, blockchain, IoT. Proficient in developing automation framework with high-quality services and developing a strategy for emerging technologies. Primary goal is to be always challenged and learning new skills to solve increasingly complex problems. She is sharing best practices among colleagues and the tech world at large. You can see her at Agile and Testing conferences.