Predicting the future isn't magic, it's Artificial Intelligence.

Understand when to use what? As a data scientist, I've always found pandas to be an excellent tool for data manipulation. Its seamless integration with NumPy allows for fast mathematical operations, making it perfect for working with smaller datasets. However, as I started working with larger datasets and had to perform data-cleaning tasks, my initial approach was to use regex in pandas. My work involves managing datasets that are in the millions and need to be processed daily. Our data orchestration tool is Airflow, but running code on Airflow for an hour wasn't an option. This is where my senior said we are using Snowflake, a NoSQL database that can run queries to clean the data for millions of data within lesser time. I used regex in Snowflake queries to clean the data, and the processing time was significantly faster than running it on pandas. While pandas is an excellent tool for smaller datasets, it's not suitable for all data manipulation tasks. I realized the import...

Some basics: linux Note: <> just used to differentiate the file for understanding : ls -list the file mkdir <folder name> - creach the file name touch <file name> - create the file code . - take to vs code vi <filename >- open the vim editor - Don’t dive deep into vim editor There are two mode Edit mode and command mode You can press Esc to enter cmd mode to give command like insert delete update and so on  i - for insert a text in a file O, o move the cursor above and below :wq - write and save the file These are the basic about vim editor but don’t spend too much time on this use other editor for faster progress and so on We can directly open the file and made the changes Vim is fun to learn cat <file name > - show what are there in that file rm -rf <file/folder name> -remove the the file or folder from the current directory cd - change the directory(where we are working and our file path) cd .. go back the previous folder in a directory Git c...

Introduction: This time, I don't want to write the hurdles in fastbook and how to overcome them because once you move to notebook 6 or 7 it shows that you have the strong mindset to overcome by yourself and finish the book. Here after we have to practise what we learn by participating in Kaggle Competitions .  Problem statement: Identify the type of disease present in paddy leaf images. Understanding the dataset Labels are given in a separate csv file, train data set contains 10 folders each containing an image of paddy disease with respect to folder name, test data contain the images we have to predict. Model Implementation Once we Explore the dataset, the next step is to build a data block for the problem. We can create using two ways they are: We may map the label of the disease name to the image by csv file. Or we may label the data according to the folder name. What I chose is mapping the label of the images with respect to folder names. Once the data loaders are ready we can ...

Notebook_5 deep dive to Image classification problem and explain the mathematical concepts in an easy understandable code way. I don't want to explain the concepts in the notebook because Jeremy and Sylvian explained in an amazing way. What I am going to discuss is that I face difficulties in some areas you might come across the same issue. That's what we are going to discuss. Once the dataset is downloaded you may come across why we are using the Path.BASE_PATH = path ,if we don't do this means we are going to write the entire path of the directory. Path.BASE_PATH = path performs the same as pathlib module. Data block is like a blueprint for the model we are going to build. While building the data block you may come across an interesting library called regex, don't get intimidated and learn everything about regex, I am sure you may get that in the deep learning journey. Go with the flow learn what you want to learn like just know what it's like $ took the last digi...

Basics of deep learning Lecture 3 is about data ethics, It's self Explanatory and Rachel’s Lecture is more than sufficient.  Lecture 4 is more about breaking the myth behind neural networks. And the basics of pytorch and how fastai api makes deep learning cool. You may come across jargon like Sigmoid, Relu, Stochastic Gradient Descent, Learning rate and so on. Their explanation is great. They explained maths with a piece of code and I completely enjoyed it. What i would suggest is play with notebook 4 and learn more things. In case you don't understand something, go through it again and again and you will catch up. If you want some help ask the question in the forum. You can practise lecture 4 by downloading the repository and going to the clean folder, there is only code not with a prose. It's the best place to practise what you have learned till now. The motto is, If you are in doubt, run the code .

Hi Friends, What i am so excited about notebook2 is we are going to deploy a model, which helps to understand the flow of deep learning. I am going to discuss the problem I faced, may be you may face in your journey. First and foremost thing is you may not be able to get the bing api. If you are getting it, It's ok but if you face any difficulties means use DuckDuck go api which comes very much handy. Once you downloaded the dataset Everything worked fine until the deployment part.  What I would suggest is if you are using the colab like me means you may face difficulties in using viola. What I will suggest is to go for Gradio which is much easier. With an few line of code you may built a amazing application. My project is about classication of great ape(Chimpanzee, Bonobos and Gorilla) eg: In case you may face any difficulties go through my notebook . Feel free to ask the doubts.

  If you are new to deep learning or stucked like me in the middle of your journey to Artificial Intelligence.  There is no good source as compared to fastai. I like the practical approach of the fastai and the teaching of the Jeremy's Top Down approach. I don't want to write a blog about what are the concepts in lecture 1 because there are so many people in the fastai forum who are wonderful in explaining the concepts. If you have any doubt ask in the forum, There is no such thing as a good question or bad question everyone in the community is helping each other out. No one is going to judge you. I am writing this because you may also face the same problem like me and it may help you in the journey. For each notebook I am going to share my experience and this is about Notebook Lecture 1(Intro). I strongly suggest you watch Jeremy's lectures before going through the notebook. First oscillation in my mind is which platform I should use for the lecture whether it is google c...

Let me tell you something, I was studying machine learning and deep learning for 7 month. In this journey, Whatever i learned it gives me the feeling of emptyness. During the journey, I came across Fastai which is an high level api built on top of the Pytorch.  Many times , I started and end up of not completing the course due to various reason but this time, I was so commited to complete it. I don't want to write this blog for an expert in Fastai or an deep learning expert. This is for the person who are one steps behind me. I will be sharing my experience may be you may face in the journey. This is an Introduction, So i dont want to dump anything which myself feels annoying for the begginers. Speaking from the experience people in the fastai forum is really amazing whatever the doubt you may come across there is always some one to help you in the journey. I begin my deeplearning journey with various online courses dumping the knowledge without knowing what and how they actually u...

After learning the Sci-kit learn library, I started to think What, Where,  How should I start learning deep learning which library should go on, I find myself Tensorflow is easy to begin with because TensorFlow is developed by google and they providing a stable and extensive documentation and so on... While learning Image classification I came to know the power of GPU, Everything comes in colab which is very much handy. Main thing I learned about deeplearning is You can do anything as long as you can convert into object into Tensors (Array of numbers) Classification of 101 image of food data image data is done with EfficientnetB0 model using transfer learning. There I, learned the power of Transfer Learning. TensorflowHub providing various Pretrained model, we can use based on our need.  And there is something called Tensorboard whwere we can the evaluate the various model, we build with deeplearning. I attached the sample image of Tensorboard.  And there are so many reso...

Project Overview: Twitter Sentiment Analysis helps to understand the user experience of the online retailers based on the tweets, which is important for business development and product enhancement.This a online retailer problem so based on the tweets we are giving the rating of each customer about online shopping experience. Sentiment analysis architecture:  Approach for Twitter sentiment analysis of online retailers: Scraping the tweets: Taking the tweets with respect to online retailers such as ebay, meesho, olx india, flipkart, amazon, jio mart.  After the extraction of the data from twitter, put into the data frame with respect to retailers.  Concatenating the data frame of each retailer into a single data frame.  Converting the data frame to CSV file.  Data preparation and data cleaning: Importing the CSV file using pandas. Analyse & understand the nature of the data.  Tweets are not actual text format, it contains no text characters, writing a fu...

Unsupervised learning: Unsupervised machine learning refers to the category of machine learning techniques where models are trained on datasets without labels. Unsupervised learning generally use to discover the patterns in data and reduce high-dimensional data to fewer dimensions.  Here, I did work on some of the clustering algorithm using scikit-learn namely,  KMeans, DBScan, Hierrarchial clustering. Dimentionality reduction and manifold learning Learning the Algorithm: I Personally feel that data cleaning and Preprocessing are challanging than training the model. Once you finished those 80% of your work is done. Then you can play around with different type of machine learning algorithm. Each algorithms are effective on its own ways. I learned the clustering algorithms through the "iris" dataset in seaborn.  Lets see some of my learning phase of unsupervised learning through Visualization: KMeans: DBScan:  Hierrarchial clustering: Dimentionality reduction: Man...

Fb-prophet We can predict the time analysis using fb prophet.Prophet follows the sklearn model API. We create an instance of the Prophet class and the call its fit and predict methods. Even we can mention holidays in the analysis. In this method monthly-milk data is used.  Source code

Introduction: Decision tree in general represent the hierarchical series of binary decision. The decision tree in the machine learning works in exactly the same way and except that we let the computer figure out the optimal structure and hierarchy of decision, instead coming up the criteria manually. In this model, I took the Australian weather dataset for forecasting.  Data Preprocessing: We'll perform the following steps to prepare the dataset for training: Create a train/test/validation split. Identify input and target columns. Identify numeric and categorical columns. Impute missing values. Scale the numeric value. Encode categorical columns to one-hot vector. Data Visualization: Tree is split on the basis of gini index.   Plot is based on the important feature of Weather prediction. Hyperparamter tuning: What we observe is that training model is 99% accuracy and validation set is just above the average, which means machine is memorizing the data in order to increase the a...

Introduction: Last time what we saw is Linear regression which is helpfull in predicting the data which is in certain patern but Logistic regression helpfull in classification problem. For example, whether the person having diseases or not ,whether will rain tommorow or not. Exploraory data analysis: In this phase we usually check the rows and columns of the given data sets and describe it and check wheather the data containing missing value or not. Seperate the numerical columns and categorical columns and split the data into train,val,test test to generalize the model prediction.  Data Visualization : Here we are analysing the co-relation of data. which is helpfull in building the better model. Actually I build a logistic regression model on breast cancer detection and weather prediction which is good for the people who are beginning the carrier in data science. Data Preprocessing : Imputing: Filling the missing value with appropriate technique. Scaling: Scaling is used to reduce...

Hi, guys this is my first machine learning model using linear regression... Introduction: In this model we will predict the insurance amount to be paid by the new customer from a given datasets. Myself download the ACME insurance datasets from the github you can try any other datasets. Let's predict the insurance amount paid by the new customers. Relation between the data: Let's analysis the some of the data using visualization technique. By analyzing the data there is a strong relation between smoker and charges. Scikit learn: It is one of the powerfull machine learning library in the field of data science. In this problem we use this to calculate linear regression(y=mx+c) model, and also using OneHotEncoder to convert multiple object into calculatable values. By understanding the line is just above the cluster this is due to the outliers. This is the simple regression line once we find for single varriable we can easily do it for multiple variable using scikit-learn. Loss: On...

Hi guys, this is my first blog toward data science!!!!! Introduction: Covid-19 or Coronovirus is a zoonotic disease which means its spread among animals and humans.India is the second most populous country, let's see how its spread and affect the people and their awarness about vaccination.  Impact of Covid-19: Unprepared lockdown which almost destroys the un organised sector. Lots of people lost their livelihood due to unemployment. Central government imposition of National lockdown affect the migration labour most of them crossed thousands of kilometers just by walking. Children's education is miserably affected. Mostly private school student are afford to online resources it create ineqauality in education and it affect basic structure of Indian constitution(Art-21A) Primary and Manufacturing sector are mostly affected which leads to reduction of GDP Total number of Positive cases: Total number of death cases: Inference from the Positive cases and death cases: Despite Mahara...