🍃MongoDB
Last updated
Last updated
In this tutorial, you'll learn MongoDB and how we use Python with MongoDB. Before starting the tutorial, let's remember the difference between structured and unstructured data and why and in which situations we use MongoDB.
We recommend that you set up a MongoDB Atlas free tier cluster for this tutorial. You can follow this tutorial to get your free MongoDB Atlas Cluster.
Structured data is most often categorized as quantitative data, and it's the type of data most of us are used to working with. Think of data that fits neatly within fixed fields and columns in relational databases and spreadsheets. Examples of structured data include names, dates, addresses, credit card numbers, stock information, geolocation, and more. Structured data is highly organized and easily understood by machine language. Those working within relational databases can input, search, and manipulate structured data relatively quickly using a relational database management system (RDBMS). This is the most attractive feature of structured data. The programming language used for managing structured data is called structured query language, also known as SQL.
Unstructured data is most often categorized as qualitative data, and it cannot be processed and analyzed using conventional data tools and methods. Examples of unstructured data include text, video files, audio files, mobile activity, social media posts, satellite imagery, surveillance imagery – the list goes on and on. Unstructured data is difficult to deconstruct because it has no predefined data model, meaning it cannot be organized in relational databases. Instead, non-relational or NoSQL databases are the best fit for managing unstructured data. Another way to manage unstructured data is to have it flow into a data lake, allowing it to be in its raw, unstructured format.
We are generating data at an unprecedented pace right now. The scale and size of this data – it’s mind-boggling! Just check out these numbers:
Facebook generates four petabytes of data in just one day
Google generates twenty petabytes of data every day
Furthermore, Large Hadron Collider (27 kilometers long most powerful particle accelerator of the world) generates one petabyte of data per second. Most importantly this data is unstructured
1 Petabyte(PB) = 220 Gigabyte(GB)
Can you imagine using SQL to work with this volume of data? It’s setting yourself up for a nightmare!
SQL is a wonderful language to learn as a data scientist and it does work well when we’re dealing with structured data. But if your organization works with unstructured data, SQL databases can not fulfill the requirements. Structured databases have two major disadvantages:
Scalability: It is very difficult to scale as the database grows larger
Elasticity: Structured databases need data in a predefined format. If the data is not following the predefined format, relational databases do not store it
So how do we solve this issue? If not SQL then what?
This is where we go for unstructured databases. Among a wide range of such databases, MongoDB is widely used because of its rich query language and quick access with concepts like indexing. In short, MongoDB is best suited for managing big data. Let’s see the difference between structured and unstructured databases:
MongoDB is an open-source document-based database system. Initially released 12 years ago, MongoDB was one of the first No-SQL system on the market, and therefore quickly gained the interest of developers. It is the most popular document database system currently on the market.
Useful links and free MongoDB courses:
MongoDB Basics https://university.mongodb.com/courses/M001/about
MongoDB for Python Developers https://university.mongodb.com/courses/M220P/about
MongoDB Atlas is a fully managed cloud database developed by the same people that build MongoDB. Atlas handles all the complexity of deploying, managing, and healing your deployments on the cloud service provider of your choice (AWS, Azure, and GCP).
MongoDB Compass is a powerful GUI for querying, aggregating, and analyzing your MongoDB data in a visual environment. Compass is free to use and source available, and can be run on macOS, Windows, and Linux.
The information in MongoDB is stored in documents. Here, a document is analogous to rows in structured databases.
Each document is a collection of key-value pairs
Each key-value pair is called a field
Every document has an _id field, which uniquely identifies the documents
A document may also contain nested documents
Documents may have a varying number of fields (they can be blank as well)
These documents are stored in a collection. A collection is literally a collection of documents in MongoDB. This is analogous to tables in traditional databases.
Unlike traditional databases, the data is generally stored in a single collection in MongoDB, so there is no concept of joins (except $lookup operator, which performs left-outer-join like operation). MongoDB has the nested document instead.
PyMongo is the official MongoDB Python driver for MongoDB. We recommend you use this driver to work with MongoDB from Python. It is easy to use and offers an intuitive API for accessing databases, collections, and documents.
Objects retrieved from MongoDB through PyMongo are compatible with dictionaries and lists, so we can easily manipulate, iterate, and print them
MongoDB stores data in JSON-like documents:
Python dictionaries look like:
Read on for an overview of how to get started and deliver on the potential of this powerful combination.
PyMongo has a set of packages for Python MongoDB interaction. To install PyMongo, open the command line and type:
Tip: If you are getting “ModuleNotFoundError: No module named 'pymongo'” error, uninstall pymongo. Use pip uninstall pymongo command. Then, re-install using the installation command.
For this python mongodb tutorial, we use mongodb srv URI, so let’s install dnspython:
Now, we can use pymongo as a python mongodb library in our code with an import statement.
The first step to connect python to Atlas is MongoDB cluster setup. Next, create a file named pymongo_test_insert.py in any folder to write pymongo code. Create the mongodb client by adding the following:
Use the connection_string to create the mongoclient and get the MongoDB database connection. Change the username, password and cluster name. In this python mongodb tutorial, we will create a shopping list and add a few items. For this, we created a database 'user_shopping_list'.
Note: MongoDB doesn’t create a database until you have collections and documents in it. So, let’s create a collection next.
To create a collection, pass the collection name to the database. Make sure to have the right indentation while copying the code to your .py file.
This creates a collection named user_1_items in the user_shopping_list database.
The list_database_names command shows the names of all the available datasets:
Output:
The list_collection_names command shows the names of all the available collections:
Output:
Note: In MongoDB, a collection is not created until it gets content, so if this is your first time creating a collection, you should complete the next part (inserting documents) before you check if the database and collection exist!
For inserting many documents at once, use the pymongo insert_many() method.
Let us insert a third document without specifying the _id field. This time we add a field of data type ‘date’. To add date using pymongo, use the python dateutil package. ISODate will not work with Python, as it is a Mongo shell function.
Install the package using the following command: python -m pip install python-dateutil
Add the following to pymongo_test_insert.py:
We use the insert_one() method to insert a single document.
Open the command line and navigate to the folder where you have saved pymongo_test_insert.py. Execute the file using the python pymongo_test_insert.py command.
Let’s connect to MongoDB Atlas UI and check what we have so far. Login to your Atlas cluster and click on the collections button. On the left side, you can see the database and collection name that we created. If you click on the collection name, you can view the data as well:
The _id field is of ObjectId type by default. If we don’t specify the _id field, MongoDB generates the same. Not all fields present in one document are present in others. But MongoDB doesn’t stop you from entering data - this is the essence of a schemaless database.
If we insert item_3 again, mongodb will insert a new document, with a new _id value. But, the first two inserts will throw an error because of the _id field, the unique identifier.
The count_documents command shows the number of documents available in “user_1_items” collection and outputs 3.
Let’s view all the documents together using find(). For that we will create a separate file pymongo_test_query.py:
Open the command line and navigate to the folder where you have saved pymongo_test_query.py. Execute the file using the python pymongo_test_query.py command.
We get the list of dictionary object as the output:
The function has returned a dictionary. Let’s see the keys of this dictionary using keys():
Output:
We can view the data but the format is not all that great. So, let’s print the item names and their category:
Although Mongodb gets the entire data, we get a python ‘KeyError’ on the third document.
To handle missing data errors in python, use pandas.DataFrames. DataFrames are 2D data structures used for data processing tasks. Pymongo find() method returns dictionary objects which can be converted into a dataframe in a single line of code.
Install pandas library as:
Replace the for loop with the following code to handle KeyError in one step:
And don’t forget to comment the print(item['item_name'], item['category'])
The errors are replaced by NaN and NaT for the missing values.
We have used find to fetch data from MongoDB. We can also use the find_one function to fetch the documents. find_one fetches only one document at a time. On the other hand, find can fetch multiple documents from the MongoDB collection. But, we don’t need to fetch all the documents all the time. This is where we apply some filter conditions.
Previously we have inserted a document to the MongoDB collection with the item_name field as Egg. Let’s see how to fetch that MongoDB document using the filter condition:
Output:
Find One
To select data from a collection in MongoDB, we can use the find_one() method. The find_one() method returns the first occurrence in the selection.
Find All
To select data from a table in MongoDB, we can also use the find() method. The find() method returns all occurrences in the selection.
The first parameter of the find() method is a query object. If we use an empty query object, it selects all documents in the collection.
Note ⇒ No parameters in the find() method gives you the same result as SELECT * in MySQL.
Return Only Some Fields
The second parameter of the find() method is an object describing which fields to include in the result. This parameter is optional, and if omitted, all fields will be included in the result.
Example: collection.find({},{ "_id": 0, "name": 1, "address": 1 })
The delete_one function deletes a single document from the MongoDB collection. Previously we had inserted the document for an item named Bread. Let’s have a look at the MongoDB document inserted:
Output:
We will now delete this MongoDB document:
<pymongo.results.DeleteResult at 0x1febb0812c8>
Let’s try to fetch this document after deletion. If this document is not available in our MongoDB collection, the find_one function will return nothing.
Output: Nothing is returned.
Since we get nothing in return, it means that the MongoDB document doesn’t exist anymore.
As we saw that the insert_many function is used to insert multiple documents in MongoDB collection, delete_many is used to delete multiple documents at once. Let’s try to delete two MongoDB documents by the name field:
Output: 2 documents deleted.
Here, the deleted count stores the number of deleted MongoDB documents during the operation. The ‘$in’ is an operator in MongoDB.
We have discussed all the basic operations with examples so far. We also understood several theoretical concepts of MongoDB.
Let me share a couple of useful functions of PyMongo:
sort: The purpose of this function is to sort the documents
limit: This function limits the number of MongoDB documents fetched by the find function
There are more MongoDB functions you can check out here.
You can update a record, or document as it is called in MongoDB, by using the update_one() method. The first parameter of the update_one() method is a query object defining which document to update.
Note: If the query finds more than one record, only the first occurrence is updated.
The second parameter is an object defining the new values of the document.
Example: Change the price from "340" to "360":
The collection before update is:
Update operation:
The collection after update is:
Update Many
To update all documents that meets the criteria of the query, use the update_many() method.
Example: Update quantity of all documents where the id starts with U1IT
Update operation:
Output:
The collection after update is:
The number of documents and collections in a real-world database always keeps increasing. It can take a very long time to search for specific documents -- for example, documents that have "all-purpose flour" among their ingredients -- in a very large collection. Indexes make database search faster, efficient, and reduce the cost of querying. For example, sort, count, match, etc.
Indexes support the efficient resolution of queries. Without indexes, MongoDB must scan every document of a collection to select those documents that match the query statement. This scan is highly inefficient and require MongoDB to process a large volume of data. If an appropriate index exists for a query, the database can use the index to limit the number of documents it must inspect.
MongoDB offers a broad range of index types and features with language-specific sort orders to support complex access patterns to your data. MongoDB indexes can be created and dropped on-demand to accommodate evolving application requirements and query patterns and can be declared on any field within your documents, including fields nested within arrays.
MongoDB defines indexes at the collection level. Indexes are special data structures that store a small portion of the data set in an easy-to-traverse form. The index stores the value of a specific field or set of fields, ordered by the value of the field as specified in the index.
You can watch the following tutorial to learn how indexing works in MongoDB Compass.
There are many different types of indexes in MongoDB that can be used for indexing a collection. Let’s take a brief look at each one:
Single-Field Index: As the name implies, this is an index on a single field of a document. For this type of index, the sort order doesn’t affect how efficiently documents are queried. It can be traversed in either descending or ascending order.
Compound Index: This type of index can be created on multiple fields. For a compound index, the order that the fields are listed impacts sorting.
Multikey Index: This type of index is used to index data that has been stored in arrays.
Hashed Indexes: This kind of index is used for large collections that are sharded and spread out over several machines. The hash of the value of a field is indexed instead of the value itself.
In the following example, we’ll show a basic API call to the create_index() method. All you need to pass is a string that represents the index name as the parameter:
If you were to print out the results of this API call, it would return a string containing the index name followed by an underscore, and the integer “1” (_1). This indicates that the index will be sorted in the default ascending order.
When you use PyMongo to create an index, you can pair the index name with either a 1 or a -1 integer value to explicitly set the sort order to ascending or descending, respectively.
The syntax for this API call is a bit different from the Mongo Shell. The PyMongo API call to create_index() requires that you pass the parameters as a tuple object nested inside of a list. An example of this syntax is shown below:
Like the last example, the response returned by the method will be a string containing the field name that was indexed, followed by an underscore (_) and a -1 because we specified that the collection would be indexed in a descending order.
Use DESCENDING or ASCENDING when indexing MongoDB collections instead of -1 and 1 respectively
In our next example, we’ll make the create_index() call more readable by passing the pymongo.DESCENDING or pymongo.ASCENDING attributes instead of a -1 or 1. To do this, we need to make sure to either import all of the classes and attributes of the pymongo library:
Another option is to explicitly import them by name when we import the MongoClient class:
You can index multiple fields in a collection by passing a Python list like we did in an earlier example; however, this time we’ll add more tuple pairs to the list, resulting in a compound index:
All of the pipelines in this post will be executed against the sample_mflix database's movies collection. It contains documents that look like this:
There's a lot of data there, but we'll be focusing mainly on the _id, title, year, and cast fields.
Aggregation pipelines are executed by PyMongo using Collection's aggregate() method.
The first argument to aggregate() is a sequence of pipeline stages to be executed. Much like a query, each stage of an aggregation pipeline is a BSON document, and PyMongo will automatically convert a dict into a BSON document for you.
An aggregation pipeline operates on all of the data in a collection. Each stage in the pipeline is applied to the documents passing through, and whatever documents are emitted from one stage are passed as input to the next stage, until there are no more stages left. At this point, the documents emitted from the last stage in the pipeline are returned to the client program, in a similar way to a call to find().
Individual stages, such as $match, can act as a filter, to only pass through documents matching certain criteria. Other stage types, such as $project, $addFields, and $lookup will modify the content of individual documents as they pass through the pipeline. Finally, certain stage types, such as $group, will create an entirely new set of documents based on the documents passed into it taken as a whole. None of these stages change the data that is stored in MongoDB itself. They just change the data before returning it to your program! There is a stage, $set, which can save the results of a pipeline back into MongoDB, but we won’t cover it.
The above code will provide a global variable, a Collection object called movie_collection, which points to the movies collection in your database.
Here is some code which creates a pipeline, executes it with aggregate, and then loops through and prints the details of each movie in the results. Paste it into your program.
This pipeline has two stages. The first is a $match stage, which is similar to querying a collection with find(). It filters the documents passing through the stage based on a query. Because it's the first stage in the pipeline, its input is all of the documents in the movie collection. The query for the $match stage filters on the title field of the input documents, so the only documents that will be output from this stage will have a title of "A Star Is Born."
The second stage is a $sort stage. Only the documents for the movie "A Star Is Born" are passed to this stage, so the result will be all of the movies called "A Star Is Born," now sorted by their year field, with the oldest movie first.
Calls to aggregate() return a cursor pointing to the resulting documents. The cursor can be looped through like any other sequence. The code above loops through all of the returned documents and prints a short summary, consisting of the title, the first actor in the cast array, and the year the movie was produced.
Executing the code above results in:
Refactoring the Code
It is possible to build up whole aggregation pipelines as a single data structure, as in the example above, but it's not necessarily a good idea. Pipelines can get long and complex. For this reason, I recommend you build up each stage of your pipeline as a separate variable, and then combine the stages into a pipeline at the end, like this:
Limit the Number of Results
Imagine I wanted to obtain the most recent production of "A Star Is Born" from the movies collection. This can be thought of as three stages, executed in order:
Obtain the movie documents for "A Star Is Born."
Sort by year, descending.
Discard all but the first document.
The first stage is already the same as stage_match_title above. The second stage is the same as stage_sort_year_ascending, but with ASCENDING changed to DESCENDING. The third stage is a $limit stage. The modified and new code looks like this:
If you make the changes above and execute your code, then you should see just the following line:
Now you know how to filter, sort, and limit the contents of a collection using an aggregation pipeline. But these are just operations you can already do with find()! Why would you want to use these complex, new-fangled aggregation pipelines? Read on, to see the true power of MongoDB aggregation pipelines.
Look Up Related Data in Other Collections
There's also a collection called comments in the sample_mflix database. Documents in the comments collection look like this:
It's a comment for a movie. The second field, movie_id, corresponds to the _id value of a document in the movies collection.
So, it's a comment related to a movie!
Does MongoDB enable you to query movies and embed the related comments, like a JOIN in a relational database? Yes, it does! With the $lookup stage.
I'll show you how to obtain related documents from another collection, and embed them in the documents from your primary collection. First, create a new pipeline from scratch, and start with the following:
The stage called stage_lookup_comments is a $lookup stage. This $lookup stage will look up documents from the comments collection that have the same movie id. The matching comments will be listed as an array in a field named 'related_comments,' with an array value containing all of the comments that have this movie's '_id' value as 'movie_id.' Added a $limit stage just to ensure that there's a reasonable amount of output.
Now, execute the code.
The aggregation pipeline above will print out all of the contents of five movie documents. It's quite a lot of data, but if you look carefully, you should see that there's a new field in each document that looks like this:
If you're lucky, you may have some documents in the array, but it's unlikely, as most of the movies have no comments. Now, we’ll add some stages to match only movies which have more than two comments.
Ideally, you'd be able to add a single $match stage which obtained the length of the related_comments field and matched it against the expression { "$gt": 2 }. In this case, it's actually two steps:
Add a field (comment_count) containing the length of the related_comments field.
Match where the value of comment_count is greater than two.
Here is the code for the two stages:
The two stages go after the $lookup stage, and before the $limit 5 stage:
Now when you run the following code, you will get an output similar to:
Output:
Now you learned how to work with lookups in your pipelines, Now you’ll learn how to use the $group stage to do actual aggregation.
Grouping Documents with $group
The $group stage is one of the more difficult stages to understand, so we'll break this down slowly. Start with the following code:
Execute this code, and you should see something like this:
Each line is a document emitted from the aggregation pipeline. But you're not looking at movie documents anymore. The $group stage groups input documents by the specified _id expression and output one document for each unique _id value. In this case, the expression is $year, which means one document will be emitted for each unique value of the year field. Each document emitted can (and usually will) also contain values generated from aggregating data from the grouped documents.
Change the stage definition to the following:
This will add a movie_count field, containing the result of adding 1 for every document in the group. In other words, it counts the number of movie documents in the group. If you execute the code now, you should see something like the following:
There are a number of accumulator operators, like $sum, that allow you to summarize data from the group. If you wanted to build an array of all the movie titles in the emitted document, you could add "movie_titles": { "$push": "$title" }, to the $group stage. In that case, you would get documents that look like this:
Something you've probably noticed from the output above is that some of the years contain the "è" character. This database has some messy values in it. In this case, there's only a small handful of documents, and I think we should just remove them. Add the following two stages to only match documents with a numeric year value, and to sort the results:
Note that the $match stage is added to the start of the pipeline, and the $sort is added to the end. A general rule is that you should filter documents out early in your pipeline, so that later stages have fewer documents to deal with. It also ensures that the pipeline is more likely to be able to take advantages of any appropriate indexes assigned to the collection.
Summary
You've learned how to construct aggregation pipelines to filter, group, and join documents with other collections. You've hopefully learned that putting a $limit stage at the start of your pipeline can be useful to speed up development (but should be removed before going to production). You've also learned some basic optimization tips, like putting filtering expressions towards the start of your pipeline instead of towards the end.
As you've gone through, you'll probably have noticed that there's a ton of different stage types, operators, and accumulator operators. Learning how to use the different components of aggregation pipelines is a big part of learning to use MongoDB effectively as a developer.
END OF THE LECTURE
Properties
Structured Data
Unstructured Data
Purpose
Structured data stands for information that is highly organized, factual, and to the point.
Unstructured data doesn't have any pre-defined structure to it and comes in all its diversity of forms.
Formats
Several formats
A huge variety of formats
Data model
Pre-defined/not flexible
Not pre-defined/flexible
Storages
Data warehouses
Data lakes
Data nature
Quantitative
Qualitative
Databases
SQL
Relational databases
NoSQL
Non-relational databases
Ease of search
Easy to search
Difficult to search
Structured Databases
Unstructured Databases
Structure
Every element has the same number of attributes
Different elements can have different number of attributes.
Latency
Comparatively slower storage
Faster storage
Ease of learning
Easy to learn
Comparatively tougher to learn
Storage Volume
Not appropriate for storing Big Data
Can handle Big Data as well
Type of Data Stored
Generally textual data is stored
Any type of data can be stored (Audio, Video, Clickstream etc)
Examples
MySQL, PostgreSQL
MongoDB, RavenDB
