The Dark Art of the Excel Services Workbook Data Refresh

In my last blog post we talked about how to create an Excel Services dashboard that pulls in information from SharePoint lists using OData feeds. This post is a continuation of that, but can also be applied to other data sources your Excel workbook might be connected to. In this walkthrough we’ll be taking a look at how you can get your dashboard to refresh in the following ways:

1. User manually refreshes the workbook
2. Workbook automatically refreshes every time it’s opened
3. Workbook automatically refreshes every x minutes (even while it’s open, or in a web part)

In this walkthrough I’ll be using SharePoint 2013 on prem. If you’re going to follow along you’ll need to have access to (and some familiarity with) SharePoint Central Administration….or you’ll need to become good friends with your SharePoint administrator (They’re probably a really nice person, you should take them out to lunch). While this walkthrough is meant to be read in order, I’ve decided to break it up into section as follows for readability (hopefully)

1. Configuring the unattended service account for Excel Services
2. Configuring authentication options in your Excel workbook for manual refresh
3. Setting the cache duration - (aka why you’re not seeing your data change immediately)
4. Make your workbook refresh as soon as it’s opened (and kill that refresh warning)
5. Make your workbook refresh right before your eyes while it’s open
6. Make Excel web parts refresh right before your eyes

1 - Configuring the unattended service account for Excel Services

One of the first things we need to consider when we talk about an Excel Services workbook refreshing is:

• What account will be used to access the data that Excel is going to refresh/go get?

After all, security is kind of a big deal, particularly when it comes to data. There are several ways to approach this, but we’re going to use what is probably the most common method and set up a service account, i.e. an account that is used specifically for getting data from our data source (in this case a SharePoint list). When we want to create a service account for these purposes in SharePoint, we call it an “Unattended Service Account”. Several service applications inside of SharePoint (including Excel) support the assignment of an unattended service account to them. Once an unattended service account has been set for a service application, it will use that account every time the service application needs to go get data.

OK great, so how do we set that up? I’m glad you asked! First we need to create what’s called a “Target Application ID”.  Target Application ID’s are basically a set of credentials that we want to use for something (in our case accessing data) and they live in their own SharePoint service application called the “Secure Store”.  Once you have a Target Application ID created, you can assign it to another service application (like Excel Services) as the “Unattended Service Account”.  To configure a Target Application ID in the Secure Store, we’re going to need access to SharePoint Central Administration. If you have no idea what Central Administration is or how to get to it, now would be a good time to go get your SharePoint administrator to help you along. During the steps to follow, we will be changing settings that have to potential to break existing workbooks that may already be using these services if someone has already configured them for other purposes.

The first thing we’re going to do is manually configure our Target Application ID in the Secure Store service application. Follow these steps:

1. Open Central Administration
2. Click on “Manage service applications” under the “Application Management” heading
3. Locate the Secure Store service application and click on it (not the proxy).
   
   If you’re having problems locating your Secure Store, it’s possible your administrator named it something different. Look in the “Type” column for “Secure Store Service Application” to help you find it. If you still can’t find it, it’s possible that the service application hasn’t been created yet. If this is the case you’ll need to create it. I suggest going here and watching the video to help you along: http://technet.microsoft.com/en-us/library/ee806866(v=office.15).aspx
   
4. Click the “New” button at the top left hand corner
5. Type “BlogTestAccount” for the “Target Application ID” and then make the necessary adjustments to make your screen match the screenshot below
   
   
   
6. Click “Next”
7. Click “Next” again to accept the default “Field Name”, “Field Type” etc.
8. In the box labeled “Target Application Administrators” enter an account that you want to be able to make changes to this Target Application ID(probably your own account at very minimum).
9. In the box labeled “Members” you want to enter the users and groups who will be allowed to use this this service account. If you have specific users that are having problems refreshing your workbook later, there’s a good chance it’s because you haven’t included them here.
   
   
   
10. Click “OK”

You should now see “BlogTestAccount” in your list of Target Application ID’s.  The next thing we need to do is actually assign an account to “BlogTestAccount” for it to use when it tries to get new data. To do that, follow these steps:

1. Place a check in the Target Application ID box next to “BlogTest Account”
2. In the ribbon at the top of the screen, click on the “Set” button in the “Credentials” group (not the set button in the permissions group)
3. Type the user name (including domain) and password for an account that has access to the data you’re going to be refreshing (aka your SharePoint list). Be sure to type this correctly, as this screen will not validate your account other than confirming that the passwords match each other
   
    
   
4. Click “OK”

Alright now that we’ve got our new Target Application ID all ready to go, we need to decide if we want Excel Services to use it as the default account every time it tries to go get data. Alternatively you can manually configure each of your workbooks individually to use a Target Application ID (we’ll cover how to do that a little later). If you want to have Excel use your “BlogTestAccount” Target Application ID as its unattended service account, follow these steps:

1. Open Central Administration
2. Click on “Manage service applications” under the “Application Management” heading
3. Locate the Excel service application and click on it (not the proxy).
   
   If you’re having problems locating Excel, it’s possible your administrator named it something different. Look in the “Type” column for “Excel Services Application Web Service Application” to help you find it. If you still can’t find it, it’s possible that the service application hasn’t been created yet. If this is the case you’ll need to create it. I suggest going here and watching the video to help you along: http://technet.microsoft.com/en-us/library/jj219698(v=office.15).aspx
   
4. Click on “Global Settings”
5. Scroll down to the very bottom of the “External Data” section and select “Use an existing Unattended Service Account”
6. In the box labeled “Target Application ID” type the name of the Target Application ID we just created “BlogTestAccount”
   
   
   
7. Click ”OK”

Congrats! We’ve finished configuring our unattended service account. Now it’s time to put it to some good use.

2 - Configuring authentication options in your Excel workbook

We’re not going to cover how you create an Excel workbook that connects to a SharePoint list using OData in this walkthrough, because I’ve already covered that here: http://tavislovell.com/create-a-dashboard-from-your-sharepoint-list-data-using-excel-services-and-the-excel-web-access-web-part/

So if you haven’t already read that post, go do so now and create yourself an Excel workbook. If you’ve already read that post, let’s get down to business and talk a little bit about authentication. With your Excel workbook open, follow these steps to get to the authentication options:

1. In the ribbon at the top, click on the “Data” tab
2. In the “Connections” group of the data tab, click the “Connections” button
   
   
   
3. Highlight the connection in your workbook, and then click the “Properties” button
   
   
   
4. Click on the “Definition” tab of the resulting Connection properties window
5. Click on the “Authentication Settings” button at the bottom of the window
   
   

At this point you should be staring directly into the eye of the beast (aka looking at the “Excel Services Authentication Settings” screen as shown below).

Let’s have a little talk about this screen, for it is here where I’ve seen many a man stumble and fall on their quest for Excel workbooks that refresh. There are only 3 options, but we must choose wisely. Let’s cover them all though.

OPTION 1 - “Use the authenticated user’s account”: In this scenario when the Excel workbook tries to refresh itself, it’s going to try to get the new data by pretending it’s you and using your account. HOWEVER, there’s a good chance it is going to fail at that task unless you have Kerberos setup in your environment. With that said, we’re going assume that you do not have Kerberos setup, so this option is of no use to us.

OPTION 2 - “Use a stored account”: This is a viable way for us to get the job done. Remember when we created our Target Application ID (BlogTestAccount) in the first section of this walkthrough? Well if we wanted to use that specific account to go get our data when the workbook refreshes, we would simply select this options and then type in “BlogTestAccount”. Feel free to use this option.

OPTION 3 - “None”: With this option, when the workbook refreshes it’s going to use the Target Application ID that was set as the unattended service account for the Excel Services service application (we also did this in the first section of this walkthrough). So this is also a perfectly good option to use assuming you were following along.

To finish this section simply choose option 2 or 3 as described above and then click “ok” back to the workbook and then save your workbook.

**NOTE** for whatever reason I’ve seen the changes above sometimes revert back to option 1 when using OData as a data source. For some reason this seems to happen almost without fail when I set these options as I’m creating a new data connection. After you save your workbook, I highly suggest going back into these settings and double checking that they’re still set appropriately.

At this point you should be able to upload your workbook to SharePoint, open it and manually refresh it using the data tab at the top. Don’t be surprised if you update the SharePoint list with a few new values, then manually refresh your Excel workbook only to see it’s not getting the new data. That’s because your data is being cached, and we’ll talk about that in the next section. The main goal here is that you’re not getting any errors when you try to refresh the workbook, and if that’s the case you’re in good shape. If you get a warning about the refresh, that’s also ok, we’ll deal with that in a bit as well. Now let’s take a look at that caching problem of ours.

3 - Setting the cache duration - (aka why you’re not seeing your data changes)

So you want to see the data in your workbook refresh on the spot, and always get the most up to date data whenever you manually refresh. Well, we could do that, but the cache is really there for a reason. It’s protecting your resources so they don’t grind to a halt if a bunch of people are constantly hitting refresh. The default cache for the refresh is 5 minutes, and personally I don’t think that’s all that long to wait. However if you wanted to adjust the amount of time that data is cached, here’s how you would do it:

1. Open Central Administration
2. Click on “Manage service applications” under the “Application Management” heading
3. Locate the Excel service application and click on it (not the proxy)
4. Click on “Trusted File Locations”
5. You should have one entry with an address of “http://”. Click on it
6. Scroll down to the bottom until you get to the “External Data” group
7. Look for the heading “External Data Cache Lifetime” as shown below

As you can see there are two different types of refresh you can configure the cache for, and they’re each measured in seconds (300 seconds works out to 5 minutes). Thus far we’ve only talked about manual refreshing, so if you’d like to change that value simply change it in the “Manual refresh” textbox and then click OK at the bottom of the screen. In a bit we’ll be taking a look at how you can have your workbooks automatically refresh every time they’re opened, or even while someone has it open and is currently looking at it. To change that cache value, simply change the value listed in the “Automatic refresh (periodic / on-open” textbox. If you’re on a dev farm with not many users, feel free to set the cache values to zero if you want to test stuff out, but in a production environment remember that the cache is your friend.

4 - Make your workbook refresh as soon as it’s opened (and kill that refresh warning)

At some point you might grow tired of having to manually refresh your workbook every time you open it. Can’t we just make the refresh happen auto-magically? Yes we can, and here’s how:

1. Open your workbook in Excel
2. In the ribbon at the top, click on the “Data” tab
3. In the “Connections” group of the data tab click the “Connections” button
   
   
   
4. Highlight the connection in your workbook, and then click the “Properties” button
   
   
   
5. Place a checkmark in the box labeled “Refresh data when opening the file”
   
   
   
6. Click OK
7. Close the “Workbook Connections” window
8. Save your workbook
9. Re-upload your workbook to SharePoint

Now every time someone opens the workbook it will refresh (although the caching described in section 3 still applies). Note that having your workbook manually refresh every time it’s opened isn’t always a great idea, particularly for workbooks with long refresh times.  

By default Excel services will prompt the user with a warning any time it’s about to run a query.  If you’ve set up your workbook to refresh on open, that warning will be the first thing you see every time you open the workbook, and it can get mighty old dismissing it over and over again.  If you’d like to fashion a pair of cement shoes for that warning here’s how you might go about it:

1. Open Central Administration
2. Click on “Manage service applications” under the “Application Management” heading
3. Locate the Excel service application and click on it (not the proxy)
4. Click on “Trusted File Locations”
5. You should have one entry with an address of “http://”. Click on it
6. Scroll down to the bottom until you get to the “External Data” group
7. Look for the heading “Warn on Refresh” and uncheck “Refresh warning enabled” as shown below
   
   
   
8. Click “OK” at the bottom of the screen

Laugh maniacally as that warning sleeps with the fishes.

5 - Make your workbook refresh right before your eyes while it’s open

Occasionally  I have someone who wants to create a dashboard in Excel, and they want to put it up on a big screen TV somewhere and have it live update in “real time”. Well, I dunno about “real time” but when most people say “real time” what they REALLY mean is “near real time”…and that we can do. But before we do, we should probably have a little discussion about what is and isn’t practical. The shortest refresh period you can set for an Excel workbook out of the box is 1 minute. So once every minute Excel is going to go out and re-query data (the caching from section 3 still applies here…go read that if you skipped it). Now this can create quite a workload for SharePoint, and whatever data source it’s pulling from. Some workbooks might not even finish refreshing in a minute. So….don’t set all your workbooks to refresh every minute. Find out how long the refresh even takes by doing some manual refreshes, and remember the amount of data is probably going to grow over time, so refreshing might start taking longer in the future. Then find out how “real time” the dashboard REALLY needs to be from a practical perspective. The less real time, the less workload it is on all the systems involved. That said, let’s make some magic happen:

1. Open your workbook in Excel
2. In the ribbon at the top, click on the “Data” tab
3. In the “Connections” group of the data tab click the “Connections” button
   
   
   
4. Highlight the connection in your workbook, and then click the “Properties” button
   
   
   
5. Place a checkmark in the box labeled “Refresh every”
6. Set the number of minutes Excel will wait between each refresh
   
   
   
7. Click OK
8. Close the “Workbook Connections” window
9. Save your workbook
10. Re-upload your workbook to SharePoint

That should do it, you’re dashboarding like a pro now! You should be able to have your Excel dashboard open, then open a new tab and change/add/delete a value or two in the SharePoint list it’s pulling from, then watch in awe as your dashboard updates whenever the next refresh hits (remember cache settings still apply, see section 3 for details)

6 - Make Excel web parts refresh right before your eyes

Welcome to final section on refreshing in Excel. You’re probably just about ready for this walkthrough to be over so I’ll make this fast. I covered how to display your Excel workbook items in web parts in my last blog post. There’s really not much involved in getting any of the refreshing techniques we’ve covered so far to work inside of the Excel web part other than making sure a few checkboxes are selected (and I believe they are by default). I’m not going to go into details about how to edit your web part etc, you can check my last blog post for those instructions, but here’s what you’ll want to look for if refreshing is not working in your web part for some reason:

1. Edit your web part so that the Excel Web Access menu is open
2. If you want manual refresh to be available, locate the “Toolbar Menu Commands” heading and make sure the “Refresh Selected Connection, Refresh All Connections” option is checked. As a side note, even if the “Type of Toolbar” is set to none, users will still be able to right click the web part and then select refresh
3. If you want to make the automatic timed refresh work, located the “Navigation and Interactivity” heading and make sure the “Periodically Refresh if Enabled in Workbook” option is checked

BOOM! That’s it! Hopefully at this point you’re feeling confident enough to start practicing the dark art of the Excel Services workbook data refresh on your own. My fingers are starting to feel pretty tired from all of this typing (pretty sure this is my longest post yet). Let me know how things work out for you in the comments and thanks for reading.