IE favorites

This article provides source code in order to display the Internet Explorer favorites in a proper order. The need for it is that, although the favorites are well listed in the user profile's Favorites folder, they don't retain the order in which Internet Explorer displays them.

For that reason, when you switch to either a total web browser alternative, like Mozilla Firebird that is able to import the IE favorites, or when you switch to a IE-based browser like MyIE2, then you face that issue.

That issue is troublesome since as a user, one tends to visually memorize the locations of favorites in the tree of folders and actual links. Switching to a web browser displaying those favorites a different way is just like displaying them randomly, it hurts.

It easily refrains the will to use that particular non-IE browser. This exactly happened to me with AvantBrowser : I remember the favorites menu was both alphabetically sorted with no way to change that, and in addition was slower to use. Not surprisingly, I switched back in just a matter of days, never got used to the new imposed behaviors.

Technically speaking, why is it that no browser seem to display the tree of favorites in a consistent manner? The remainder of this article explains that.

Where are IE favorites actually stored?

Against all odds, the tree of favorites are stored both physically and logically. Because they are stored in separate locations, they have to sync up all the time, this proves to be a source of loack of synchronization there of between the two, for instance a folder exists in one place and not in the other, but neither this issue will be scrutinized in this article, nor is the goal of the source code to help IE store its favorites!

Physically, the favorites are stored in the Favorites user profile's folder. If your name is Brian, then that folder is (by default) C: \ Documents and Settings \ Brian \ Favorites. This folder, just like any folder has files and subfolders. Both mimic the context menus and context submenus displayed in IE with the favorites menu and favorites sidebar. Files are internet shortcuts, that is files with a hidden .url extension. If you right-click an internet shortcut, and edit it with notepad, then you'll get its actual target, plus a few other options. The action of resolving an internet shortcut in order to find the target url is managed by a OS-level COM component that is part of the shell.

As you have guessed from the previous explanations, there is no ordering among internet shortcuts, as well as folders. In fact, just like any other hard drive folder displayed in Windows Explorer, you can visually sort the columns according to the names, dates, sizes, and so on, but this sorting is not reflected in any way by Internet Explorer.

The way Internet Explorer knows about the order to favorites is in a logical place, the registry. Even at this point, the favorites are logically stored in two keys. The first key, HKEY_CURRENT_USER \ Software \ Microsoft \ Windows \ CurrentVersion \ Explorer \ MenuOrder \ Favorites, manages the state and ordering of IE favorites displayed from inside Internet Explorer. This other key, HKEY_CURRENT_USER \ Software \ Microsoft \ Windows \ CurrentVersion \ Explorer \ MenuOrder \ Start Menu, however manages the state and ordering of IE favorites displayed from the Windows start menu, Favorites shortcut. (for some reason, those who have designed Windows have thought that users would take advantage of a different ordering of favorites).

The remainder of the explanations stick with the first key, HKEY_CURRENT_USER \ Software \ Microsoft \ Windows \ CurrentVersion \ Explorer \ MenuOrder \ Favorites. If you open up the registry and expand the key, you'll notice a load of subkeys on the left side, and an Order value on the right hand-side for each key. Quite naturally, the subkeys reflect the tree of folders, while each Order value reflects all actual favorites state and ordering at a given level. This Order value is a binary value and stores content in an undocumented format. In the goal of understanding how the ordering is stored and managed, we'll have to hack this structure. The next section explains the resulting investigations, and won't go in further details about the techniques involved to hack the structure, ensure the consistency, and so on.

Hacking the internal structure

The internal structure inside the Order value of each subkey is as follow :

Where the header is structured as follow :

And where each record is structured as follow :

The tables above tell us, for each entry, if the entry is an actual internet shortcut (url) or a folder, and whether its description is encoded with the local charset or with unicode. When dealing with an internet shortcut, the description is the name of the internet shortcut, with the .url suffix added to it, as a way to name the actual physical file.

Note that since parent relations are not expressed, the description is inherently top-down and it's up to the client application to "create" and hold the parent/children relationship between Order values. Of course, because of the tree nature of the registry subkeys, an inherent structure is easy to guess.

The structure is read thanks to this code :

// definition of the storage for a favorite
typedef struct _Favorite
{
  TCHAR szPath[MAX_PATH];
  BOOL bType; // TRUE = url, FALSE = folder
  TCHAR szUrl[INTERNET_MAX_URL_LENGTH];
} Favorite;


BOOL CFavorites::ExtractFavorites(HKEY hParentKey,
                                  LPTSTR szRegKeyPath, // current logical path
                                  LPTSTR szFilePath, // current physical path
                                  Favorite** ppFavorites, // forwards the structures being created
                                  BOOL bResolveUrl, // should we resolve the .url?
                                  BYTE* buffer, // content of an Order value
                                  DWORD nLength) // size of an Order value
{
  if (!buffer || nLength == 0) 
    return FALSE;

  DWORD* pdword_buffer = (DWORD*) buffer;

  // read header
  DWORD nTotalLength = pdword_buffer[2];
  DWORD nRecords = pdword_buffer[4];

  if (nRecords == 0) 
    return TRUE;

  *ppFavorites = new Favorite[nRecords + 1];
  if (!*ppFavorites)
    return FALSE; // out of memory

  DWORD i;
  BYTE* p = buffer + 20;

  for (i = 0; i < nRecords+1; i++)
  {
    (*ppFavorites)[i].szPath[0] = 0;
    (*ppFavorites)[i].szUrl[0] = 0;
  }	
	
  for (i = 0; i < nRecords; i++)
  {

    DWORD length = *( (DWORD*)p );
    DWORD sortedindex = *( (DWORD*)(p+4) );
    if (sortedindex == 0xFFFFFFFB) // -5 = folder not sorted, all urls are as listed as is
      sortedindex = i;
    WORD  key1 = *( (WORD*)(p+8) );
    WORD  key2 = *( (WORD*)(p+10) );
    DWORD d1 = *( (DWORD*)(p+12) );
    DWORD id = *( (DWORD*)(p+16) );
    WORD  stringlength = *( (WORD*)(p+20) );

    BOOL bUrl = (key2 & 0x0002) ? TRUE : FALSE; // url or folder
    BOOL bUnicode = (key2 & 0x0004) ? TRUE : FALSE; // unicode encoding or codepage encoding
		
    if (sortedindex < nRecords)
    {
      (*ppFavorites)[sortedindex].bType = bUrl;


      TCHAR szUrlPath[MAX_PATH];
      _tcscpy(szUrlPath, szFilePath);
      _tcscat(szUrlPath, "\\");

      if (bUnicode)
      {
        int cch = ::WideCharToMultiByte(CP_ACP, 0, (OLECHAR*)( p + 22 ), -1, 
                                        NULL, 0, NULL, NULL); 
        ::WideCharToMultiByte(CP_ACP, 0, (OLECHAR*)( p + 22 ), -1, 
                              (*ppFavorites)[sortedindex].szPath, cch, NULL, NULL); 
      }
      else
      {
        _tcscpy((*ppFavorites)[sortedindex].szPath, (char*)( p + 22 ));
      }

      _tcscat(szUrlPath, (*ppFavorites)[sortedindex].szPath);
      _tcscpy((*ppFavorites)[sortedindex].szPath, szUrlPath);

      // should we resolve the url?
      if (bUrl && bResolveUrl)
      {
        LPTSTR szUrl = NULL;
        ResolveInternetShortcut(szUrlPath, &szUrl);

        if (szUrl)
          _tcscpy((*ppFavorites)[sortedindex].szUrl, szUrl);

        m_pMalloc->Free(szUrl);
      }

    }

    p += length;

    if (p >= buffer + nLength)
      break; // avoid access violation
  } // for i

  return TRUE;
}


// use the shell url parser to resolve what's the .url file actually targets
void CFavorites::ResolveInternetShortcut(LPTSTR szPath, LPTSTR* lpszURL)
{
  *lpszURL = NULL;  // Assume failure
	
  IUniformResourceLocator* pUrlLink = NULL;
  HRESULT hr = ::CoCreateInstance(CLSID_InternetShortcut, NULL, CLSCTX_INPROC_SERVER,
                                 IID_IUniformResourceLocator, (void**)&pUrlLink);
  if (FAILED(hr))
    return;

  IPersistFile* pPersistFile = NULL;
	
  hr = pUrlLink->QueryInterface(IID_IPersistFile, (LPVOID*)&pPersistFile);
  if (SUCCEEDED(hr))
  {
    // Ensure that the string is Unicode. 
    WCHAR wsz[MAX_PATH];  
    ::MultiByteToWideChar(CP_ACP, 0, szPath, -1, wsz, MAX_PATH);
		
    // Load the Internet Shortcut from persistent storage.
    hr = pPersistFile->Load(wsz, STGM_READ);
    if (SUCCEEDED(hr))
      hr = pUrlLink->GetURL(lpszURL);
		
    pPersistFile->Release();
  }
	
  pUrlLink->Release();
}

In order to simplify the sequential or recursive use of this method, a few methods are added as part of what we shall call a general API for the extraction of favorites :

BOOL ExtractFavorites(Favorite** ppFavorites,
                      BOOL bResolveUrl); // root level
BOOL ExtractFavorites(LPTSTR szRegKeyPath,
                      LPTSTR szFilePath,
                      Favorite** ppFavorites,
                      BOOL bResolveUrl); // another level
BOOL ExtractFavorites(HKEY hParentKey,
                      LPTSTR szRegKeyPath,
                      LPTSTR szFilePath,
                      Favorite** ppFavorites,
                      BOOL bResolveUrl); // another level
BOOL ExtractFavorites(HKEY hParentKey, 
                      LPTSTR szRegKeyPath,
                      LPTSTR szFilePath,
                      Favorite** ppFavorites,
                      BOOL bResolveUrl,
                      BYTE* buffer,
                      DWORD nLength); // another level

A sample built on that new API

The source code uses the API to build a tree control that gets incrementally filled. Anytime the user expands a folder node, an API method call fills the underlying level (only one level). Of course, this is done only once. The trick is to use a dummy node anytime a folder node is created as a way to show the [+] handle and allow the expand event to occur.

The entire UI logic is found in the CFavExtractDlg class :

void CFavextractDlg::OnTreeItemExpanding(NMHDR *pnmh, LRESULT *pResult)
{
  CTreeCtrl* pTreeCtrl = (CTreeCtrl*) GetDlgItem(IDC_FAVTREE);
  if (!pTreeCtrl) return;

  NM_TREEVIEW *pnmtv=(NM_TREEVIEW *)pnmh;
  HTREEITEM hExpandingItem = pnmtv->itemNew.hItem;
  if (!hExpandingItem) return;

  if (pnmtv->action != TVE_EXPAND) // is being expanded
    return;

  // is there a <dummy> child?
  HTREEITEM hChildItem = pTreeCtrl->GetChildItem(hExpandingItem);
  if (!hChildItem)
    return;
	
  CString szChildName = pTreeCtrl->GetItemText(hChildItem);
  if ( szChildName.CompareNoCase(DUMMYITEMLABEL)==0  ) // there is 
  {
    // remove this child
    pTreeCtrl->DeleteItem(hChildItem);
		
    if (hExpandingItem && pTreeCtrl->GetParentItem(hExpandingItem))
    {
      LockWindowUpdate(); // CWnd member

      SetCursor(LoadCursor(NULL, IDC_WAIT)); // hourglass

      CString szPath = GetPathFromNode(hExpandingItem);
      AddChildren(szPath, hExpandingItem);

      SetCursor(LoadCursor (NULL, IDC_ARROW)); // back to normal cursor

      UnlockWindowUpdate();  // CWnd member
    }
  }
}


// helpers ///////////////////////////////////////////////////////////


CString CFavextractDlg::GetPathFromNode(HTREEITEM hItem)
{
  CTreeCtrl* pTreeCtrl = (CTreeCtrl*) GetDlgItem(IDC_FAVTREE);
  if (!pTreeCtrl) return _T("");

  if (hItem == NULL || pTreeCtrl->GetParentItem(hItem) == NULL) 
    return _T("");

  CString szPath;
  CString szCurKeyname;
  HTREEITEM hNextItem;

  while ( (hNextItem = pTreeCtrl->GetParentItem(hItem)) )
  {
    szCurKeyname = pTreeCtrl->GetItemText(hItem);
    if ( !szPath.IsEmpty() )
      szCurKeyname += "\\"; 

    szPath =  szCurKeyname + szPath;
    hItem = hNextItem; 
  } // end while

  return szPath;
}


// AddChildren
//
// Fill the tree ctrl with one level of favorites
//
// parameters : szPartialPath : empty is this is the root
//                              path of the form Favorites\\Devnews
//              hParent         parent tree item
//
void CFavextractDlg::AddChildren(/*in*/CString &szPartialPath, HTREEITEM hParent)
{
  CTreeCtrl* pTreeCtrl = (CTreeCtrl*) GetDlgItem(IDC_FAVTREE);
  if (!pTreeCtrl) return;

  Favorite* favs = NULL;

  CFavorites f;
	

  TCHAR szRegPath[MAX_PATH];
  TCHAR szFilePath[MAX_PATH];
  f.GetFavoritesRegPath(szRegPath);
  f.GetFavoritesFilePath(szFilePath);
  CString szFullRegPath = szRegPath;
  CString szFullFilePath = szFilePath;
  if ( !szPartialPath.IsEmpty() )
  {
    szFullRegPath += "\\" + szPartialPath;
    szFullFilePath += "\\" + szPartialPath;
  }
  else
  {
    // create tree root item
    if (!hParent)
    {
      long i = szFullFilePath.ReverseFind('\\');
      hParent = pTreeCtrl->InsertItem( 
         szFullFilePath.Right( szFullFilePath.GetLength() - (i+1) ), 
         FOLDERICON, FOLDERICON );
    }

  }

  f.ExtractFavorites(szFullRegPath.GetBuffer(0),
                     szFullFilePath.GetBuffer(0),
                     &favs, 
                     TRUE /*resolve url*/);

  if (favs)
  {
    Favorite* p = favs;
    while ( 1 )
    {
      if (p->szPath[0] == 0)
        break; // no need to go further

      HTREEITEM hItem;

      // short filename
      CString s = _tcsrchr(p->szPath, '\\');
      if ( !s.IsEmpty() )
      {
        s = s.Right( s.GetLength() - 1);

        if ( p->bType ) // url
        {
          if (s.Right(4).CompareNoCase(".url") == 0) // remove .url suffix
            s = s.Left( s.GetLength() - 4);

          hItem = pTreeCtrl->InsertItem(s.GetBuffer(0),
                                        URLICON, 
                                        URLICON,
                                        hParent);
        }
        else // folder
        {
          hItem = pTreeCtrl->InsertItem(s.GetBuffer(0),
                                        FOLDERICON, 
                                        FOLDERICON,
                                        hParent);

          // if this is a folder, insert a dummy item only to ensure we have a [+] sign
          pTreeCtrl->InsertItem(DUMMYITEMLABEL,
                                URLICON,
                                URLICON,
                                hItem);
        }

      }

      p++;
    }
  }

  delete [] favs;
}

Summary :

• favorites.h, favorites.cpp is the actual favorite extractor
• favExtractDlg.h, favExtractDlg.cpp is a sample MFC dialog with a tree control
• the extractor relies on shell32 version 4.71 for the internet shortcut resolver. This version came with Internet Explorer 4.0 in all Windows OS versions. Thanks to this, no further constraint should prevent the API from working regardless the OS and browser version.

History

• Feb 4, 2004 - first posting.
• June 20, 2004 - update to support Windows XP as well.