A comprehensive MongoDB troubleshooting guide

As far as scalable document databases go, none are as versatile and developer-focused as MongoDB. Whether you want to store a small number of JSON documents, perform large-scale transactions, run powerful aggregations and transformations, or implement stream processing, MongoDB offers a quick and easy solution.

However, since it plays a central role in many modern IT infrastructures, prompt troubleshooting of its issues is crucial to avoid system-wide bottlenecks or disruptions. In this guide, we will discuss the most common MongoDB issues, including those related to startup, connectivity, scalability, configuration, replication, and locking contention.

What is MongoDB?

MongoDB was initially released as a NoSQL data store, but has since evolved into a comprehensive developer data platform that simplifies building complex data-driven applications. The developer-friendly Mongo API offers a convenient way to interact with the database via a wide array of supported programming languages. Here are some stand-out MongoDB features

• Flexible schema: The document-based structure allows for dynamic schema evolution to accommodate changes in data layout without downtime or complex migrations.
• Rich query language: The MongoDB query language supports aggregation pipelines, geospatial queries, full-text capabilities, and more.
• Built for scalability and high availability: Out of the box, MongoDB offers seamless horizontal scaling via sharding. It also comes with customizable built-in mechanisms for replication and automatic failover.
• Generative AI: MongoDB has introduced several built-in features in its latest releases that fuel AI use cases, including vector search and generative AI integrations.
• Transactions: Unlike other NoSQL stores, MongoDB natively supports multi-document ACID transactions, which allows for complex data operations with atomicity, consistency, isolation, and durability guarantees.

Server startup and connection problems

Let’s consider a few issues you may face when starting or connecting with a MongoDB server.

Issue # 1 – MongoDB won’t start

Problem: You ran the mongod command but the server failed to start.

Detection: On your terminal, you see error messages indicating that the server failed to start or nothing at all.

Troubleshooting:

• Start by verifying whether the process is already running. You can use the ps aux | grep mongod command to do so. If it’s running but not in a healthy state, you can shut it down using mongod –shutdown
• If the server isn’t already up, review the errors on the console and the MongoDB log files more closely. The path of the log file is specified in the MongoDB configuration file or passed as input to the mongod command. If you don’t see anything useful in the logs, try increasing the verbosity level and restarting.
• Confirm that the user with which you are running mongod has adequate permissions to access the necessary directories and files. If not, use chown or chmod to adjust the permissions as necessary.
• Double-check your mongod.conf for any syntax errors or incorrect paths.
• Check system resource utilization, including CPU, memory, and storage space, to ensure that MongoDB has sufficient resources to start and operate.
• Ensure that the port you are using (27017 is the default) is free and not in use by another application. To verify, run this command: netstat -nlp | grep 27017.
• Another reason for a MongoDB instance not starting is disk corruption or missing data files. In such cases, you should create a backup of your data files and then start MongoDB with this command: mongod --dbpath /data/db --repair

Issue # 2 – Connection denied or timeout errors

Problem: You are unable to connect to the MongoDB server from a command line tool or a client application.

Detection: You receive errors like “Connection refused” or “Connection timed out”.

Troubleshooting:

• Check whether the MongoDB service is in a healthy state. You can use commands like service mongod status for this purpose.
• Check the MongoDB file logs for additional information related to the errors.
• Confirm that the port you are using to connect matches the one defined in the MongoDB configuration file. You can run the command netstat -nlp | grep mongod to see the port MongoDB is configured to run with.
• Double check your entire connection string to ensure that it has the correct combination of hostname, port, and any required authentication details.
• If connecting from a remote machine, rule out any connectivity related issues by using the ping and telnet commands. Moreover, if a firewall is configured, ensure that it contains the required rules to allow the client–server communication via the relevant port.
• If TLS is enabled, verify that all the relevant parameters in the configuration file are properly configured, including certificateSelector, mode, allowConnectionsWithoutCertificates, and certificateKeyFile.

Issue # 3 – Permission denied errors

Problem: You are able to connect, but unable to perform any operations on the database.

Detection: You encounter errors that indicate you have insufficient permissions for specific operations or collections.

Troubleshooting:

• Double-check if you are connecting to the database via the right user.
• Verify that your user has the necessary roles and privileges for the actions you are performing. You can use db.getRoles() for this purpose. More often than not, “permission denied” errors happen due to misconfigured user roles.
• If your user has the relevant permissions but you are unable to interact with a specific collection, check whether that collection has any custom permissions to restrict access.
• If the above steps don’t work, restart MongoDB with a higher log verbosity to potentially get more clues about the issue.

MongoDB configuration issues

Now, let’s focus on some common MongoDB misconfigurations; how to detect and resolve them.

Misconfiguration # 1 – Improper memory configuration

Problem: Configuring insufficient memory for MongoDB can lead to performance bottlenecks and even server crashes.

Detection: You may be experiencing sluggish query response times, frequent server restarts, or OOM (Out of Memory) errors in logs.

Troubleshooting:

• Check your mongod.conf for the storage.wiredTiger.engineConfig.cacheSizeGB setting. Adjust it based on available RAM and expected workload.
• Tweak other memory-related settings with the prefix storage.wiredTiger as per your specific needs.
• Use tools like free on Linux to track memory usage in real-time.
• For large data sets or high write volumes, consider sharding your database to distribute the load across multiple servers.

Misconfiguration # 2 – Inappropriate value for max connections

Problem: Setting an inappropriate value for max connections can lead to connection rejections and client application timeouts.

Detection:

• You may be observing sudden spikes in connection errors and timeouts whenever MongoDB is under high load.
• You may be seeing errors in the logs indicating that connections are being rejected because the limit has been reached.

Troubleshooting:

• Check your mongod.conf file for the current value of the net.maxIncomingConnections setting.
• Use purpose-built monitoring tools, like Site24x7’s MongoDB monitoring tool, to track and analyze your application's expected concurrent connections and workload to determine a suitable value. Consider factors like peak usage period and query complexity.

Note: A high maxIncomingConnections value alone doesn't guarantee optimal performance. Ensure that your server hardware and other configuration settings can support the increased connection load.

Misconfiguration # 3 – Missing or inefficient indexes

Problem: A lack of proper indexes or using inefficient ones can significantly impact query performance.

Detection:

• You may observe slow queries in your profiler or application logs.
• You may see index-related warnings or errors in the MongoDB logs.

Troubleshooting:

• Use the db.collection.explain() method to analyze query execution plans and identify opportunities for index optimization. For example, if you have a collection named Users that contains the names and ages of users, you can use the following query to retrieve useful information:

  db.users.find({ age: { $gt: 25 } }).explain("executionStats")

  Sample users table:

  { "_id": 1, "name": "John", "age": 32 } 
  { "_id": 2, "name": "Charles", "age": 29 }  
  { "_id": 3, "name": "Alex", "age": 32} 

• Create indexes on frequently queried fields or fields that are used in sorting or filtering operations. This will boost overall performance.
• Evaluate if existing indexes are still relevant and performing as expected. Remove unused or inefficient ones. To retrieve all indexes for a collection, use this command: (replace users with the name of your collection)

  db.users.getIndexes() 

MongoDB performance bottlenecks

Now, we will examine some common MongoDB performance bottlenecks and how you can go about resolving them:

Bottleneck # 1 – Long-running queries

Problem: Some queries are taking too long to complete, causing apps to slow down and affecting user experience.

Detection: Your monitoring tool or profiler is reporting spikes in average query execution times.

Troubleshooting:

• Use the db.currentOp() method to identify all the running queries on your system. The output will show all the relevant details, including operation type, query plan, the duration, and client connection responsible for the operation.
• Ensure that proper indexes are in place for all commonly used filters and sort criteria. Analyze the explain plan of relevant queries to see if indexes are being used as expected.
• Refactor queries to avoid unnecessary operations like full collection scans or inefficient filtering conditions. Use built-in query options like skip or limit to fetch only the relevant data.

Bottleneck # 2 – Excessive resource utilization

Problem: High CPU, memory, or disk usage can lead to performance bottlenecks and even crashes.

Detection: Your monitoring tool is repeatedly raising alerts for excessive resource utilization.

Troubleshooting:

• Identify and optimize resource-intensive queries or operations.
• If resource utilization is the bottleneck, increase system resources (CPU, memory, and storage) to handle increased workload or a higher number of concurrent connections.
• Consider sharding or partitioning large data sets to distribute query load across multiple nodes.
• Consider implementing resource management strategies, like connection pooling, query throttling, or workload isolation, to improve overall system stability.

Bottleneck # 3 – Concurrency and lock contentions

Problem: High write volume or complex queries can lead to lock contention, which in turn causes performance degradation and increased wait times.

Detection:

• Logs show lock contention-related errors.
• Queries get stuck in the waiting phase.

Troubleshooting:

• Execute the db.serverStatus() command and look for the globalLock and locks sections. Analyze acquireCount, acquireWaitCount, currentQueue, and other similar parameters to identify the root cause of lock contention.
• Use the output of the db.currentOp() command to identify the queries that have the waitingForLock flag set to true. Dig deep into the competing queries and explore avenues to reduce contention.
• Ensure that relevant indexes are present to facilitate faster lookups and reduce locking.
• If you have high-write workloads, consider distributing data across multiple servers to reduce lock contention on individual shards.

Bottleneck # 4 – Issues related to storage space

Problem: Insufficient disk space or slow disk I/O can also lead to bottlenecks and potential data loss. Potential reasons for this issue can be growing data sets, inefficient storage configuration, or hardware limitations.

Detection:

• You are running out of space on your MongoDB server.
• Your monitoring tool is reporting slow disk I/O.

Troubleshooting:

• Regularly clean up old or archived data that is no longer used.
• If applicable, store compressed data in your collections to reduce the overall storage footprint. For example, you can create a new collection with zlib compression via this command:

  db.createCollection('test', {storageEngine: {wiredTiger: {configString: 'block_compressor=zlib'}}}) 

• Regularly analyze the size of all your collections using the db.users.stats().size command (where users is the name of the collection) to monitor data growth and optimize accordingly.
• If you are still experiencing storage-related issues despite performing the above steps, consider upgrading your storage capacity or exploring cloud-based storage solutions.

MongoDB replication-related issues

Here are some common issues that you may face if you have replication configured on your MongoDB infrastructure:

Issue # 1 – Replication lag

Problem: Secondaries are lagging behind the primary, impacting potential failover readiness.

Detection:

• Replica members of the deployment are showing stale data.
• MongoDB logs are reporting errors related to replica lag.

Troubleshooting:

• Use the rs.status() command to check replication status in real-time. Look for relevant fields like optime, optimeDurable, uptime, state, and health to get a better idea of the problem.
• Ensure that there are no network or bandwidth-related bottlenecks between any members of the replica set.
• Insufficient disk space on a secondary can also halt replication. Monitor disk usage and free up space if needed.
• Specify a suitable value for the replication.oplogSizeMB parameter in the MongoDB configuration file based on your replication needs to avoid excessive oplog growth.

Issue # 2 – Data divergence

Problem: Secondary replicas have deviated from the primary replica due to replication failures or inconsistencies. This is known as data divergence and can lead to synchronization issues across the replica set.

Detection:

• There are visible differences (such as in the count or contents of the documents) in the data stored on primary and secondary replicas.
• MongoDB logs show warnings or errors related to replication.

Troubleshooting:

• Dig deep into the file logs across all nodes to identify the root cause of the divergence.
• Rule out network connectivity or bandwidth-related issues.
• If applicable, perform a resynchronization (resync) of affected secondary replicas by removing and re-adding them to the replica set. You can do this by using the rs.remove() and rs.add() commands.
• Verify that the oplog size (replication.oplogSizeMB) and retention settings are appropriately configured to capture all write operations and prevent oplog overflow.

Issue # 3 – Election failures

Problem: The primary replica has failed but a new primary couldn’t be elected due to configuration issues or network partitions.

Detection:

• Since the primary is unavailable and is the only recipient of write operations, you may notice that no replication is taking its place.
• It’s also possible for multiple primaries to be up, causing data inconsistencies across the cluster.

Troubleshooting:

• Verify network connectivity and configurations for all replica set members to ensure they can participate in elections.
• Check for network partitions or split-brain scenarios that may be preventing members from reaching a consensus.
• Review the replica set configuration settings, like the priority and votes properties, to ensure the desired behavior during elections.
• After ruling out network or configuration related issues, consider manually forcing a new primary election using the rs.stepDown() command on the current primary replica. The command converts the current primary into a secondary, kickstarting the election process.
• In complex scenarios, consider using rs.reconfig() to manually reconfigure the replica set.

MongoDB best practices

Finally, let’s explore some MongoDB best practices that can help keep an instance running smoothly for extended periods:

• Regularly update your MongoDB server and drivers to receive the latest security patches, performance improvements, and new features.
• Incorporate native MongoDB tools like db.currentOp(), db.stats(), mongostat, and mongotop into your administrative workflows to streamline database management and gain real-time insights into performance.
• Create well-designed indexes on frequently used fields to expedite query execution and optimize data retrieval.
• Optimize your queries to avoid full collection scans, unnecessary joins, and inefficient filtering conditions. Leverage explain plans to understand how a query is being processed and identify opportunities for optimization.
• Formulate a comprehensive backup strategy to protect against data loss due to accidental deletion, hardware failures, or cyberattacks.
• Consider encrypting sensitive data. MongoDB supports encryption both at rest and in transit.
• Regularly monitor the health and performance of your MongoDB infrastructure using dedicated tools like the MongoDB Monitoring tool. Site24x7 helps you to track key metrics in real-time, including those related to memory, replica sets, shards, and connections.

Conclusion

MongoDB is a robust and flexible data platform that fuels business use cases across several industries. To keep it running smoothly, it’s crucial to solve its issues and bottlenecks promptly and effectively. In this guide, we dissected MongoDB issues related to startup, connectivity, configurations, replication, and performance; we hope you found it useful.