PSConfig /SharePoint Configuration wizard runs longer when upgrading the servers.

During production farm (multi servers environment) SP1 upgrade,  I noticed that the psconfig was running significantly longer. I expected that the first run would be the only one that took very long time, but in my case the subsequent runs on the other servers took on average 25 minutes to finish. As we know that the first run will update the SharePoint databases so, it makes senses that it will take longer time in proportion to the number of the databases. But subsequent runs should only take around 5 minutes each.

I noticed that my config wizard seems paused at certain times, then I checked the upgrade log in the 14 hive\LOGS, which I found that during that apparent paused the log spit out alot of this below message, and they are a lot, as this site is in Mysite DB which has hundreds of site collections.

[PSCONFIG] [SPContentDatabaseSequence] [DEBUG] [3/3/2012 10:10:24 PM]: Site with Id = 961cfd1e-5cf8-4e40-8756-0032a517119b is contained in dictSitesNeedUpgrade, but is not added. Possible sitemap conflicts. Could not connect to http://localhost:32843/SecurityTokenServiceApplication/securitytoken.svc. TCP error code 10061: No connection could be made because the target machine actively refused it 127.0.0.1:32843.

Looking at that message, it seems that the psconfig tried to communicate with the Token Service, which obviously shut down during the process because the IIS service W3SVC (World Web Pulbhisng Service) is stopped.

So I looked at the Central Admin > Upgrade and Migration > Review Databases Status and I found that any content databases for any site collections that use Claim Based Authentication have Status field saying Database is up to date, but some sites are not completely upgraded. It became clear to me why the psconfig run much longer during the servers upgrade, because it tries to upgrade any claim based site collections by calling unavailable Security Token Services. It seems the psconfig try to call this service several times before it spits out the error message and it worsened in my case as there are hundreds of site collections of this type.

So my solution is before running the psconfig  to upgrade the servers (after successfully upgrade the database by running the psconfig at the first time), I run powershell commands to upgrade the databases that have status not completely upgraded. as mentioned in previous paragraph. This will work as the Token Service will be available. Run below powershell script for each db once, where the db_name is the ‘not fully upgraded’ database name. After run this command, my configuration wizard only runs for approximately 5 minutes in each server.

  $ct = Get-SPContentDatabase -Identity <db_name>
  Upgrade-SPContentDatabase -id  $ct.id