Internet of Things (IoT) has attracted considerable attention in recent years due to its potential of interconnecting a large number of heterogeneous wireless devices. However, it is usually challenging to provide reliable and efficient random access control when massive IoT devices are trying to access the network simultaneously. In this article, we investigate methods to introduce intelligent random access management for a massive cellular IoT network to reduce access latency and access failures. Toward this end, we introduce two novel frameworks, namely, local device selection (LDS) and intelligent preamble selection (IPS). LDS enables local communication between neighboring devices to provide cluster-wide cooperative congestion control, which leads to a better distribution of the access intensity under bursty traffics. Taking advantage of the capability of reinforcement learning in developing cooperative multiagent policies, IPS is introduced to enable the optimization of the preamble selection policy in each IoT clusters. To handle the exponentially growing action space in IPS, we design a novel reinforcement learning structure, named branching actor-critic, to ensure that the output size of the underlying neural networks only grows linearly with the number of action dimensions. Simulation results indicate that the introduced mechanism achieves much lower access delays with fewer access failures in various realistic scenarios of interests.