Thought I’d put this out there since it took me a little bit of time to diagnose and resolve.

Migrating from Exchange 2003 to 2010. Installed 2010 and put the Client Access role on a different machine than all the other roles. OWA worked, ActiveSync worked, but you couldn’t connect using Outlook within the office without wonderfully descriptive errors relating to basic Outlook/Exchange connectivity, the same ones you would get if your server was offline, Address Book Service was stopped, etc,…

Outlook connects to Exchange using RPC so I checked the RPC Client Access log (C:\program files\Exchange Server\V14\Logging\RPC Client Access by default) and found it full of errors like this:

6/19/2011 7:08:20 AM, currently Mounted”,”RopHandler: Logon: [RopExecutionException] Misconfiguration: Mailbox DB LONG-HEX-ID hosted on /o=First Organization/ou=Exchange Administrative Group (MY ADMIN GROUP ID)/cn=Configuration/cn=Servers/cn=MY-SERVER-NAME is pointing to RpcClientAccess on this server, /o=First Organization/ou=Exchange Administrative Group (ADMIN GROUP ID AGAIN)/cn=Configuration/cn=Servers/cn=SAME-SERVER, which doesn’t have a ClientAccess role installed. Error code = NoServerSupport”

And there’s your problem. It was looking for client access where client access did not exist. I verified by popping open the Exchange shell and running a few commands.

Get-MailboxDatabase

FL

This showed me all the info about my database, including the Identity, which I needed for the next step, and confirmed that RpcClientAccess was in fact set to the wrong server. The fix was easy.

Set-MailboxDatabase -identity “Mailbox Database ID” -RPCClientAccessServer CORRECT-INTERNAL-DNS-NAME

Worked immediately after changing this. I’m not quite sure why this was necessary. My hunch was that I initially installed the Client Access role on the single machine, then uninstalled it, then installed it on a different machine. There might be a way to change this from the console, too, but this was easy enough.

Hope it helps you out sometime.

I’m sitting in a burger place in Cherry Hill, NJ. When I walked in, the girl at the counter told me that they can only do cash right now because their credit cards system is down. Their entire register would be inoperable if someone hadn’t discovered that they could disconnect the CAT5 to send it into an offline processing mode where it will just queue transactions until later.

The reason is awesome. Apparently, the company that runs their registers is down because their antivirus installed either new definitions or an application update that is preventing something from working, I’m guessing a false positive. They’ve been down since last night. Even better, this took down everyone using these registers, which is apparently a ton of other area businesses. Not sure if it’s national or not but either way, it is hilarious. I hate the antivirus development industry.

I don’t even know what to say at this point. Back in February, I wrote this, an entry about a longstanding bug that makes your system BSOD if you put BESR and BE on the same machine. Well guess what? In the new version of Symantec System Recovery 2011, the bug is still there. And how do I know? Because it just BSOD’ed by office’s server as I was trying to run a backup. There’s something ironic about your backup software causing the need to restore from a backup. We were lucky, no corrupt databases or anything, but seriously, WTF? When I exclaimed to my co-workers about what happened, one of them laughed and threw up his hands to exclaim, “How is this company still in business?” How indeed.

This is a dumb one that took far longer than I would have liked.

SBS 2008. OWA was spitting out HTTP Error 500.19 – Internal Server Error

The requested page cannot be accessed because the related configuration data for the page is invalid.

It then listed information, owaauth.dll was mentioned somewhere. Should start taking screen shots of these… It also referenced a line of web.config from program files\windows small business server\bin\webapp\sbs web applications, a line containing only . The event log showed an error relating to the DLL as well:

The Module DLL C:\Program Files\Microsoft\Exchange Server\ClientAccess\Owa\auth\owaauth.dll failed to load. The data is the error.

And this one:

Could not load all ISAPI filters for site ‘SBS WEB APPLICATIONS’. Therefore site startup aborted.

The error on the website complained of something being locked.

Searches all talked about 32bit mode vs 64bit, unlocking strings, resetting IWAM and IUSR accounts (though they don’t seem to apply to SBS 2008?), recreating your OWA folders… I did all that for hours with no luck. In the end, it was permissions: owaauth.dll had Authenticated Users granted Read and not Read & Execute. I discovered this by comparing the file to another server. The whole OWA folder was jacked. IISRESET and we were back up.

The moral here is to always remember to check your basics first. Two of my co-workers looked at this for well over an hour before it was passed to me and I won’t say how long I spent working on it before I found this. If the error says that a file can’t be loaded, maybe it actually means that the file can’t be loaded.

It is midnight and I promised myself I would go to sleep an hour ago. That promise seems to be the one I am the worst at keeping, right up there with flossing, exercising every day, regular oil changes, and basically all of those things that we’re told responsible adults do when in reality they exist just to make us feel like losers. Or something like that. Regardless, it’s midnight and though I want to sleep, I also want to get this out while it’s still somewhat fresh in my mind.

I have a love-hate relationship with Symantec. Funny, I feel like I made that statement before… Oh, that’s right, I did back in February when BESR was BSOD-ing my servers. Well it’s still true and no matter how hard it hits me, I keep coming back with hope that next time, this very next time, things will be better. Of course, they never are; those sneaking suspicions you have in the beginning of a relationship, those red flags telling you to get out because it’s just not going to work — they’re always right.

I have been wearing rose-colored glasses when looking at Symantec Backup Exec for years. When it works, it is great. Lots of features, an interface that does make sense once you spend a few minutes with it, support for granular restore of all sorts of things, works well with all kinds of different media, not really prone to crashing… As long as you aren’t trying to backup something that makes use of its Advanced Open File Agent. Oh, and Exchange might give you weird errors. And Hyper-V? If it doesn’t give you VSS errors, you’re still limited to full backups of your VHDs only because it doesn’t operate at a block-level.

The problems with Backup Exec are staggering. We have installed it on dozens upon dozens of servers and the amount of hand-holding required to make it do its job consistently just blows my mind. To Symantec’s credit, their tech support staff is usually very knowledgeable and will try their best to resolve and educate, but nothing changes the fact that it is the rule, not the exception, for VSS errors to just appear out of nowhere, require a reboot, and then everything will just work. The knowledgebase articles linked from the event log are often completely unrelated to your issue, resolutions to simple issues are elusive… What the hell is going on? While our most simple servers, usually stand-alone DC/File/Print servers without Exchange, Hyper-V, or SQL seem to work consistently, get a multi-server network, add in a few roles, maybe a USB drive, and you are fucked.

So with the taste of Backup Exec 11 through 2010 R2 (I haven’t played with R3 yet) still reminding me of something between orange juice + toothpaste and cat urine, I was delighted to start using Backup Exec System Recovery 2010 and the new Symantec System Recovery 2011. As I described in my post back in February, this application, which is Norton Ghost with a different name and some extra features intended for enterprise users, simply works. Always. I guess because it works at a block-level, there is less that can go wrong? Whatever it is, install is a breeze, setup is a breeze, backup and restore are fast, granular restore works well, technical issues (aside from that BSOD thing…) have not been bad, the P2V/V2P options are great, and the price is fantastic.

The pricing for the virtual edition of Symantec System Recovery 2011 is very attractive. For something like $2000, you can install on a physical host and unlimited Hyper-V VMs. This appealed to me and a client whose Backup Exec-protected Hyper-V cluster was a shitstorm of negativity. While we provide help desk support and try to stay attentive, we are not there full-time and cannot afford a backup software that requires constant hand-holding so it does its goddamn job. I downloaded and installed SSR 2011’s trial, a 350 MB-ish application. Install was easy, configured a backup, configured offsite backup, created my first image — awesome. And then I decided to install the management “solution.”

And that’s when things got bad.

The Management “Solution” is something like 1.8GB in a single ISO. I downloaded it simultaneously to my client’s machine, a Server 2008 R2 Enterprise Edition box and my company’s server, same version of the OS. I made sure to download what was indicated as the 64-bit edition. A short time later, I extracted the ZIP, ran the Setup executable on my client’s machine and… got an error that it wasn’t a valid 32-bit application. What the fuck? I tried the browser application in the same directory, same error. I tried it on my server — different but similar error. Immediately. We were not off to a good start. I dug through the folders a bit and buried somewhere in there I found the installer for the 64-bit version of the program, which I discovered was the Symantec Installer or something along those lines. I installed the installer, fired it up, and about 90 minutes later had installed the prerequisites for the management utility which included SQL Express and the IIS role, then the various components that made up the software: Altiris Management Console 7.1, the installer for SSR 2011, the SSR 2011 remote agent, and the SSR 2011 Management Console.

I’m going to first whine about this installation process. The fact that I had to jump through hoops is one thing, the fact that it took about 90 minutes to install this software is another; the fact that I am installing a very sophisticated management application to remote control NORTON fucking GHOST: SERVER EDITION is too absurd for words. The fragmentation of Symantec’s product lines is unreal. Symantec Endpoint Protection has its own console that’s pretty cool, Backup Exec has its own management utility, Brightmail Security looks exactly like SEP (and correct me if I’m wrong but I think the two integrate?), and then I am installing the largest, most cumbersome, complex utility to control what is their most straightforward, functional, “set-it-and-forget-it” business backup app? Are you kidding me? We’re going to come back to this.

I’m going to skip the rest of the story and cut right to the facts.

After the installation was finished, I was unable to login to the SSR 2011 console because the shortcut the application created was formatted incorrectly. Yes, immediately after the installation, I couldn’t get into the application, the huge application that took forever to install, that made me hunt for its executable. The reason was idiotic: you manage this software through a web browser. There is an option to manage the application through SSL and I set the port to the default, 443. After troubleshooting for far too long, I discovered that the URL looked like this:

http://localhost:443/Altiris/blahblahblah

But… 443 isn’t HTTP, it’s HTTPS. I modified the shortcut to be https://localhost/altiris/blahblahblah and it loaded right up.

I spent some time looking at the documentation. The interface is clunky, figuring out exactly how to find a machine and deploy the agent and then the software took far too long and didn’t seem to want to discover anything other than the local host. It didn’t want to deploy the SSR 2011 Remote Agent and it was telling me that the application was not installed, something I attributed to the installation of the software first and then the management console. If there is a next time, which I hope there isn’t, I will do the management console first. After some fighting, I got the remote agent installed locally.

The SSR 2011 application would not accept my license key. I mean, it would kind of accept it after a 2 minute+ hang, it would say that it was OK, but then I’d check Help -> About it and it would say trial. The management console said the same thing.

Changes to the backup policies were complicated to apply. Changes to just about everything were hard to apply.

With all this complaining, I want to make it clear that figuring stuff out is my goddamn job. I get paid pretty well for it, I consider myself pretty good at it, I really like doing it. I am confident that if I had sat there long enough, fighting with this software, I would have made more sense of it. I am sure that it is powerful and has all kinds of great features. That’s nice and all but the fact is, a simple, straightforward application like Norton Ghost should not have a complex management utility that plain doesn’t work. We’re going to come back to this when I’m done describing how shitty the software is.

I uninstalled SSR 2011 manually so I could push it down from the console, thinking that might correct my license issue and other quirks. Well, the software wouldn’t install from the remote agent. The Windows Event Log showed an IIS error relating to ASP.NET permissions, something I was able to find someone else describing in a thread on Symantec’s forum when they were beta testing Altiris 7.1. Again, what the fuck? That’s it? I messed around with it a bit, tired, hungry, and frustrated. It got worse, I uninstalled and reinstalled, it failed. I threw my keyboard, sent a defeated email to my co-workers who were telling me that the app would suck from the get-go, and went home. As soon as I have a moment, I will call the client, tell him that the software was junk, recommend AppAssure (Kaseya integration, competitive price, fantastic features, NOT MADE BY SYMANTEC), and manually rip out the pieces of these products that did not want to come out on their own.

That is my SSR Management Solution Experience but that is not the whole of my rant.

Growing up with Windows, I think that many of us learned to expect that things wouldn’t work. There are these paradigms that we except as laws, things like “new software will have bugs,” “sometimes you just need to reboot to fix a weird error,” “powerful software is inherently complicated.” It’s kind of like living in a third-world country that doesn’t have clean running water without ever knowing how we live here in America. You just don’t know so you think it’s normal.

Linux changed that for me. I remember how shocked I was when I started getting into basic web app development and configured my server. I needed to install some prerequisites and it told me what they were and asked if I wanted to download and install them. I hit Y and Enter, it installed them, and that was that. My server can stay up for weeks, months, without ever needing a reboot. Things don’t crash, they don’t freeze. My backups just work. Apple has proven this to us as well with their desktop OS and iOS. So why is it still acceptable for Windows applications made by large corporations and intended to provide crucial protection for businesses so inconsistent and unreliable? Is it Windows? I don’t know. Windows 7 works pretty well and so does Server 2008 R2, though we still have to reboot after making most system configuration changes

At the risk of talking about things that I don’t understand, I feel as though big companies feel as though they have the IT industry by the balls when it comes to certain applications. They know that because the expectations are so low, they only have to be just a little bit better than the other guys to be ahead of the curve. Backup Exec performs so inconsistently that when a new version is released and it STILL has the same stupid VSS problems as the last three versions, we will be a little frustrated but shrug it off — that’s just what VSS does. Except it’s not, as shown by so many other backup utilities. Symantec System Recovery and its management “solution” are new, so we can expect a few bugs right away. I mean, that’s why we don’t install Windows Updates or Service Packs until everyone else has tested them first, right? I call bullshit.

I find that there is a double standard when it comes to treatment of end users. Consumers are babied because there are more of them and they are seen as stupid because, well, most people really are stupid. Because of that, Symantec made sure that Norton Ghost was simple, efficient, and easy. When they ported it to servers, it was already all cleaned up. The same goes for the client-side version of SEP: clean, easy, unobtrusive. But IT professionals? They know that we are used to shit just not working. They know that we expect to have to struggle because we’re supposed to be cool and patient, we fix things and deal with problems for a living so we expect there to be rough edges… and I think it is absolute garbage.

An anecdote to illustrate:

Google. I like Google Apps. Features, price, presentation. A great alternative for small businesses to Exchange. We have been selling it so I signed us up to be resellers.

The Google Apps Reseller signup process works like this: you apply. You prove to them that you can sell the product by recruiting something like 20 users for them, basically as a test. When you’ve done that, you submit your information and you wait. And you wait. And you wait. There is no indicator of where your company is in the queue, no way of checking on your status, no automated, “Hey, we haven’t forgotten about you, we’re still processing applications and we appreciate your patience” emails. Their website has no contact information other than a reseller signup forum which, as you can guess, is filled with people asking “What is my status?” Everyone waits over a month, no exceptions. If you complain, they escalate you.

In my case, I complained publicly on their board that I not only had to wait a month with no response but their attitude was disrespectful and un-Google-like. For a company built on friendly, simple, functional technology to ask potential salespeople to sell their products on good faith and then accept a response of, “Good job! Now don’t call us, we’ll call you!” is insulting. I sold the product on good faith and I deserve for my time to be respected. I should not have to bitch publicly to find out that they were having trouble getting information about my company because of some typo or something, some intern should write a little interface that says my status. Quick and easy. And you know what? A Google spokesperson responded immediately, apologized, agreed, said that his hands were tied and they would love to provide more information but their development staff was focused on adding features for resellers. The signup process was low on the priorities list.

That sucked but it’s over; my point is that these double-standards show what these giant corporations think they can get away with. Symantec is far worse, with their fragmented, barely-functional, over-engineered, bloated applications.

I resent the idea that technology has to be complicated to be powerful. The problem I see is that so many companies have built industries around their software having kinship to archaic magickal arts, rituals known only to a select few who have devoted their lives to studying and practicing, that if they were to suddenly release a new version that was built for better usability, people would be out of jobs and pissed off. If Cisco’s web interfaces were suddenly more like SonicWall’s, they would lose all that money coming in from certifications and sooner or later, people would be out of jobs because there would be no need for them, the elevator and telegraph operators of the modern era. We tolerate cumbersome, clunky interfaces because that’s what we’ve always known and for some, buried within that is job security, it is profit… but it is also elitism, it also keeps powerful technology out of the hands of those who lack the funding to pay for experts with the knowhow to operate it. It ensures that people who have devoted their life to feeling superior for spending the time to learn this technology won’t have to learn new skills and stay competitive in the market.

And this takes us back to Symantec System Recovery 2011 and its Management “Solution.” At the end of the day, I was left with a management console that was the nuclear reactor to my backup software’s battery-powered flashlight. Why would software that screams for use by small businesses, priced cheaply enough to put it in the hands of just about anyone, have a management console with a confusing installation process, a ton of extra server roles and prerequisites required, a BROKEN shortcut immediately after installation, errors in the event viewer, and a clunky-as-all-hell user interface? If I can setup a Hyper-V cluster and write troubleshooting blog posts that generate a shit-ton of traffic to my site every single day, I can figure out the quirks of a management console and make it work… but why should I have to? Why am I expected to? Because we have an enterprise IT culture that tolerates shitty software, a corporation that buys smaller companies and develops parallel versions of redundant software, and a corporate software development culture that knows we have low standards so they have absolutely no reason to change.