Role: Deep Learning Engineer | Machine Learning Engineer

Company: DQLabs https://www.intellectyx.com/ | https://www.dqlabs.ai/

Project: Augmented Data Quality Platform

Developed and implemented advanced machine learning and natural language processing techniques to enhance DQLabs.ai, an augmented data quality platform. The platform enables organizations to measure, monitor, remediate, and improve data quality across various types of data, leveraging ML and self-learning capabilities.

Technologies and Libraries:

• Programming Language: Python
• NLP Libraries: NLTK, Gensim
• Deep Learning Frameworks: Keras, PyTorch
• Word Embeddings: GloVe, BERT/SBERT
• Data Processing: PySpark

Key Accomplishments:

1. Developed a Deep Learning/Neural Network/NLP model to determine the semantic type of provided data.
   • Implemented state-of-the-art algorithms, including Sherlock and Sato, for accurate semantic type detection.
   • Utilized Sherlock, a deep neural network-based single column type detection algorithm, leveraging multi-feature vectors such as paragraph-to-vector, word embedding feature vectors (GloVe and BERT/SBERT), bag-of-characters feature vectors, and bag-of-words feature vectors.
   • Employed Sato, a cutting-edge algorithm for semantic type detection, to enhance the model's performance and robustness.
2. Built a system that identifies the semantic type (e.g., Name, SSN, Phone) for each column in a dataset.
   • Designed and implemented a flexible and scalable architecture to handle diverse column names and underlying data patterns.
   • Developed a mechanism to incorporate training sample datasets as new patterns and data types are uncovered, enabling continuous improvement of the semantic type detection model.
3. Integrated NLP and deep learning libraries and frameworks, including NLTK, Gensim, Keras, and PyTorch, into the Python codebase.
   • Leveraged the power of these libraries to preprocess and transform textual data, extract meaningful features, and build robust deep learning models.
   • Utilized NLTK for tasks such as tokenization, stemming, and lemmatization, Gensim for topic modeling and document similarity, and Keras and PyTorch for building and training deep neural networks.
4. Achieved an accuracy rate of 90% and above for each column type semantic detection.
   • Conducted extensive experiments and fine-tuned the models to optimize their performance and generalization ability.
   • Employed techniques such as cross-validation, hyperparameter tuning, and regularization to prevent overfitting and ensure high accuracy across diverse datasets.
5. Developed the ability to build an API on top of the semantic type detection system to integrate with PySpark.
   • Designed and implemented a RESTful API that exposes the semantic type detection functionality to external systems and applications.
   • Ensured seamless integration with PySpark, enabling the processing of large-scale datasets and leveraging the power of distributed computing.

Throughout the project, I demonstrated strong expertise in natural language processing, deep learning, and big data technologies. By leveraging state-of-the-art algorithms like Sherlock and Sato, along with popular NLP and deep learning libraries, I successfully developed a highly accurate and scalable semantic type detection system. This system significantly enhances the data quality management capabilities of DQLabs.ai, empowering organizations to gain valuable insights and make data-driven decisions with confidence.