An Exercise In Optimization

Sometimes when developing code you can spend to much time trying to optimize code that does not need to be optimized, especially if it makes the code harder to understand (often negatively referred to as "premature optimization").  However, sometimes optimization can be very worthwhile.

A quick example to share.  I was working on a segment of code last week that was taking a long time to run; on the order of 5+ minutes for something that seemed like it should be much quicker.  The code made sense when looked at, but it definitely felt as though this one might be worth spending time to optimize.

Upon closer inspection, I realized that a key index was missing from the database lookup.  This code has recently changed from processing a flat file to pulling information from a Sqlite database.  The table it was having to query was large, and the column it was querying against wasn't part of an index.  Upon adding the index, the run time of the code dropped from over 5 minutes down to 31 seconds.

All done!  Or not.  Once you are there, it's worth making sure you are not overlooking other possible issues.  I realized it was also querying one other table that, although it was smaller, was suffering from the same issue.  After adding an appropriate index for that table as well, the run time had dropped to 15 seconds.

But that's not all!  Upon a closer review of the code, I realized it was repeatedly looking up data with the same parameters in a lot of cases, so I added an in memory cache to cache that data as needed throughout the processing.  This dropped the total run time down to 5 seconds.

And then I realized I had also left out from the caching another lookup that was only needed when populating the cache.  After moving that last lookup to the code that utilized the cache, the run time had dropped again down to 14 milliseconds.

From 5+ minutes to 14 milliseconds.

I wasn't aware I could make it that fast, but by being mindful of what seems reasonable and being careful in my review of how the processing was operating (and not exclusively looking at just the database or just one segment of code), I was able to dramatically improve it.  Sometimes optimization can be very worthwhile.  Just use your best judgement on where it seems worthwhile, and where it would just complicate the code.

Lost in a Wormhole

The blackness of space ravages my mind.

It has been almost a year since that fateful day. I had found new wealth in shipping goods between regions -- my small contribution to serving the people of the Empire. I finally had the resources to buy the finer ships of the galaxy, including the largest freighters to support my trading federation. But I wasn't content. Null space was a mess, overrun by savages lacking any code of conduct. High security space was a bore. And with pirates as active as ever in high security shipping lanes, my freighters were locked up in space port with most of my shipments still using the sexy but limited deep space transports.

That is when I saw the shimmer. It drew me in a first glance. It drew me in with a dance of colors that ebbed and flowed that seemed somehow both random and orderly at the same time. It was like no wormhole I had encountered before. On any other day I might have turned away to avoid the unseen dangers, but this was no ordinary day. Dissatisfied with known space and mesmerized by the wormhole to worlds unknown, I charged forth with all the bluster of a drunk sailor on a mission.

That was almost a year ago. I've seen amazing worlds and unbelievable stellar phenomena, but I am lost among the stars. I yearn to go home without any idea how to get there. At my wits end, coasting on the fumes of a starving fusion reactor, I suddely saw something familiar. A glimmer of light. And a glimmer of hope. First, a moment of disbelief. Then, of heighened excitement, as I burst from my chair and spilt my drink. The way home! That's it! Then, finally, a feeling of dread. What husk of a shell of what was already a decaying Empire would I find when I get back? Would anything be the same? Would anything be left worth returning home for?

My sudden dread came about so suddenly as to leave me in shock. At that moment, I remembered the Mystic who spoke to me many years ago. Such an odd encounter at the time. "Heart and mind", she told me. Alone, the heart can leave you in chaos, and the mind can make you cold. Only together can they find balance, and lead you out of the dark. "You will find yourself in the dark one day", she said. "Let your heart be the Captain, and your mind the Navigator."

I think I understand now. My heart yearns to reconnect with my people. To try to make things right. My mind can help me get there, and keep me steady on the journey. It's time. I'm going home.

Handling EVE ESI Authentication with Spring Boot

Continuing on from my prior post on reading from the EVE ESI API with Java, today I provide some details on handing the EVE ESI authentication flow with Spring Boot.

With ESI, they expect your application to be a web application, whether it is one or not.  If it's not, things get more complicated.  The problem stems from their new flow where you are supposed to redirect to them for EVE sign on and then they redirect back to you with a code you use to obtain access to restricted data.  This is problematic if your application isn't a web application.

If you are using Java, this problem happens to be an ideal use for Spring Boot.  With Spring Boot, configuration is mostly a breeze and an application server like Tomcat can be easily embedded in your Spring Boot application to handle the redirect and callback.

First, a quick overview of the steps you need to take, then below I provide an example Controller class for a Spring Boot application that can handle all of this.

1. Determine your IP address, pick a port, choose a callback request mapping, and build the URL for your application from all that.  (example: http://[your IP]:[port]/[request mapping]).  It is recommended you use https, but doing so will require additional setup I'm not going into here.
2. Register your application at the EVE Developers Site, using the URL from the previous step as your application callback URL.
3. Access your router and forward TCP for your port to the machine that will be running your application.
4. Create your Spring Boot application.  Maven dependencies provided below the Controller example should you find it helpful.
5. Run your application.  Following the controller flow intended for the example, you would have an index page with a link to your /launchSignOn page.  Clicking that should redirect to EVE where you sign in.  Following that, it should redirect back to your application, where your application then takes the code sent back, makes a post request to get an access token (which includes a refresh token you will want), and makes a request to get character info, and then do whatever.  If you are just trying to get a refresh token and character ID for a different application, it can just display the fresh token and character ID on the screen for your to copy down.

Example Spring Boot Controller 

@Controller
public class SampleController {

 // these variables could come from a properties file or elsewhere, shown here as constants 
 // just for the sake of this example
 private static final String SCOPE = "esi-markets.read_character_orders.v1";  // example
 private static final String MY_CLIENT_URL = "?/eveCallback";  // callback URL for your application
 private static final String MY_CLIENT_ID = "?";   // client ID for your registered EVE application
 private static final String MY_SECRET_KEY = "?";  // secret key for your registered EVE application
 private static final String STATE = "?";            // your application state, if any
 private static final String MY_REFRESH_TOKEN = "?";  // saved after first sign in for subsequent use
 
 /**
  * A start page, should you want one.  It could have links to /launchSignOn and /testRefreshToken
  * 
  * @return view page reference
  */
 @RequestMapping(value="/", method=RequestMethod.GET)
 public String startPage() {
  return "index";
 }
 
 /**
  * Begin authentication process.  This will redirect to EVE sign on which will later redirect back 
  * to /eveCallback.  The redirect URL you here should be the same as the one you entered for your
  * registered EVE application, and it should point to your callback request mapping.
  * 
  * @return redirect URL to EVE sign on
  */
 @RequestMapping(value="/launchSignOn", method=RequestMethod.GET)
 public String launchSignOn() {
  return "redirect:https://login.eveonline.com/oauth/authorize"
    + "?response_type=code"
    + "&redirect_uri=" + MY_CLIENT_URL
    + "&client_id=" + MY_CLIENT_ID
    + "&scope=" + SCOPE
    + "&state=" + STATE;
 }
 
 /**
  * EVE sign on should redirect back to here. Now you can send a post request to get an
  * access token (which also includes a refresh token you probably will want to save), and if needed,
  * you can make a request to obtain character details as well (in particular, you will probably
  * want the character ID).
  *  
  * @param model
  * @param code    the code you need to request an access token
  * @param state   your appliction state info, should match what you originally sent in the redirect
  * 
  * @return view page reference, which could display info you want to see about the token or character.
  */
 @RequestMapping(value="/eveCallback", method=RequestMethod.GET)
 public String catchCallback(ModelMap model, @RequestParam String code, @RequestParam String state) {
  // do something with your state if needed
  MoxyJsonConfig moxyJsonConfig = new MoxyJsonConfig();
  ContextResolver<MoxyJsonConfig> jsonConfigResolver = moxyJsonConfig.resolver();
  Feature basicAuth = HttpAuthenticationFeature.basic(MY_CLIENT_ID, MY_SECRET_KEY);
  Client client = ClientBuilder.newBuilder()
    .register(basicAuth)
    .register(jsonConfigResolver)
    .build();
  MultivaluedMap<String, String> formData = new MultivaluedHashMap<String, String>();
  formData.add("grant_type", "authorization_code");
  formData.add("code", code);
  AccessToken accessToken = client.target("https://login.eveonline.com/oauth/token")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .post(Entity.form(formData), new GenericType<AccessToken>(){});
  Feature bearerAuth = OAuth2ClientSupport.feature(accessToken.getAccess_token());
  client = ClientBuilder.newBuilder()
    .register(bearerAuth)
    .register(jsonConfigResolver)
    .build();
  EveCharacter character = client.target("https://login.eveonline.com/oauth/verify")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .get(new GenericType<EveCharacter>(){});
  model.put("accessToken", accessToken);
  model.put("character", character);
  return "showTokenAndCharacterId";
 }
 
 /**
  * Uses a previously stored refresh token to obtain a new access token and character information.
  * 
  * @param model
  * 
  * @return view page reference, could display whatever info you want to see about token or character.
  */
 @RequestMapping(value="/testRefreshToken", method=RequestMethod.GET)
 public String testRefreshToken(ModelMap model) {
  final MoxyJsonConfig moxyJsonConfig = new MoxyJsonConfig();
  final ContextResolver<MoxyJsonConfig> jsonConfigResolver = moxyJsonConfig.resolver();
  Feature basicAuth = HttpAuthenticationFeature.basic(MY_CLIENT_ID, MY_SECRET_KEY);
  Client client = ClientBuilder.newBuilder()
    .register(basicAuth)
    .register(jsonConfigResolver)
    .build();
  MultivaluedMap<String, String> formData = new MultivaluedHashMap<String, String>();  
  formData.add("grant_type", "refresh_token");
  formData.add("refresh_token", MY_REFRESH_TOKEN);
  AccessToken accessToken = client.target("https://login.eveonline.com/oauth/token")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .post(Entity.form(formData), new GenericType<AccessToken>(){});
  Feature bearerAuth = OAuth2ClientSupport.feature(accessToken.getAccess_token());
  client = ClientBuilder.newBuilder()
    .register(bearerAuth)
    .register(jsonConfigResolver)
    .build();
  EveCharacter character = client.target("https://login.eveonline.com/oauth/verify")
    .request(MediaType.APPLICATION_JSON_TYPE)
    .get(new GenericType<EveCharacter>(){});
  model.put("accessToken", accessToken);
  model.put("character", character);
  return "showTokenAndCharacterId";
 }
}

Example Maven Dependencies

  <parent>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-parent</artifactId>
    <version>2.0.1.RELEASE</version>
  </parent>
  <dependencies>
    <dependency>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-web</artifactId>
    </dependency>
    <dependency>
     <groupId>org.springframework.boot</groupId>
     <artifactId>spring-boot-starter-tomcat</artifactId>
    </dependency>
    <dependency>
     <groupId>org.springframework</groupId>
     <artifactId>spring-webmvc</artifactId>
    </dependency>
    <dependency>
     <groupId>org.apache.tomcat.embed</groupId>
     <artifactId>tomcat-embed-jasper</artifactId> <!-- only needed if using JSP -->
     <scope>provided</scope>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.core</groupId>
     <artifactId>jersey-client</artifactId>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.inject</groupId>
     <artifactId>jersey-hk2</artifactId>
     <version>2.26</version>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.media</groupId>
     <artifactId>jersey-media-moxy</artifactId>
     <version>2.26</version>
    </dependency>
    <dependency>
     <groupId>org.glassfish.jersey.security</groupId>
     <artifactId>oauth2-client</artifactId>
     <version>2.26</version>
    </dependency>
  </dependencies>

Calling EVE OpenAPI with Java

If you are into Java and want a quick example of a way you can read data from the EVE OpenAPI, here is an example I wrote using Jersey 2.26.

import java.util.List;

import javax.ws.rs.client.Client;
import javax.ws.rs.client.ClientBuilder;
import javax.ws.rs.core.GenericType;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.ContextResolver;

import org.glassfish.jersey.moxy.json.MoxyJsonConfig;

/**
 * Example of calling the EVE OpenAPI using Jersey 2.26.  Specific Jersey components needed
 * include jersey-client 2.26, jersey-hk2 2.26, and jersey-media-moxy 2.26.
 */
public class EveOpenAPIExample {

 // variable names need to match names from the EVE OpenAPI
 private static class Region {
  private String region_id;
  private String name;
  private List<String> constellations;
  private String description;
  public String getRegion_id() {
   return region_id;
  }
  public void setRegion_id(String region_id) {
   this.region_id = region_id;
  }
  public String getName() {
   return name;
  }
  public void setName(String name) {
   this.name = name;
  }
  public List<String> getConstellations() {
   return constellations;
  }
  public void setConstellations(List<String> constellations) {
   this.constellations = constellations;
  }
  public String getDescription() {
   return description;
  }
  public void setDescription(String description) {
   this.description = description;
  }
 }
 
 public static void main(String[] args) {

  // setup the service information we will need for the example
  final String target = "https://esi.tech.ccp.is/latest/";
  final String path = "universe/regions/{regionid}/";
  final String regionId = "10000002";
  final String dataSource = "tranquility";
  
  // create the client configured to use MOXy for JSON binding
  final MoxyJsonConfig moxyJsonConfig = new MoxyJsonConfig();
  final ContextResolver jsonConfigResolver = moxyJsonConfig.resolver();
  Client client = ClientBuilder.newBuilder()
   .register(jsonConfigResolver).build();
  
  // execute example call on EVE OpenAPI to read information on a region
  Region region = client.target(target)
   .path(path)
   .resolveTemplate("regionid", regionId)
   .queryParam("datasource", dataSource)
   .request(MediaType.APPLICATION_JSON_TYPE)
   .get(new GenericType<Region>(){}); 
  
  // utilize the data
  System.out.println("Read information for region " + region.getName());
  
  System.exit(0);
 }
}

Planetary Interaction Tables

If you haven't tried it, Planetary Interaction is a great way to supplement your income, produce supplies you may need, and add some variety to your Eve experience. The key thing you need to know is what supplies can be created and what the "recipes" are for each. There are some great pages made by others that provide this information, but in my effort to host more of this information myself as well, I've provided some recipe tables here.

To find planets you are interested in close to the area you want to operate in, try using the Dotlan Planet Range tool.

1st Level Refined	Raw Resource	Barren	Gas	Ice	Lava	Oceanic	Plasma	Storm	Temp.
Bacteria	Micro Organisms	X		X		X			X
Biofuels	Carbon Compounds	X				X			X
Biomass	Planktic Colonies			X		X
Chiral Structures	Non-CS Crystals				X		X
Electrolytes	Ionic Solutions		X					X
Industrial Fibers	Autotrophs								X
Oxidizing Compound	Reactive Gas		X
Oxygen	Noble Gas		X	X				X
Plasmoids	Suspended Plasma				X		X	X
Precious Metals	Noble Metals	X					X
Proteins	Complex Organisms					X			X
Reactive Metals	Base Metals	X	X		X		X	X
Silicon	Felsic Magma				X
Toxic Metals	Heavy Metals			X	X		X
Water	Aqueous Liquids	X	X	X		X		X	X

2nd Level Refined	1st Level Refined		1st Level Refined
Biocells	Biofuels	+	Precious Metals
Construction Blocks	Reactive Metals	+	Toxic Metals
Consumer Electronics	Toxic Metals	+	Chiral Structures
Coolant	Water	+	Electrolytes
Enriched Uranium	Toxic Metals	+	Precious Metals
Fertilizer	Proteins	+	Bacteria
Gen Enhanced Livestock	Proteins	+	Biomass
Livestock	Biofuels	+	Proteins
Mechanical Parts	Reactive Metals	+	Precious Metals
Microfiber Shielding	Industrial Fibers	+	Silicon
Miniature Electronics	Silicon	+	Chiral Structures
Nanites	Reactive Metals	+	Bacteria
Oxides	Oxygen	+	Oxidizing Compound
Polyaramids	Industrial Fibers	+	Oxidizing Compound
Polytextiles	Industrial Fibers	+	Biofuels
Rocket Fuel	Electrolytes	+	Plasmoids
Silicate Glass	Silicon	+	Oxidizing Compound
Superconductors	Water	+	Plasmoids
Supertensile Plastics	Oxygen	+	Biomass
Synthetic Oil	Electrolytes	+	Oxygen
Test Cultures	Water	+	Bacteria
Transmitter	Chiral Structures	+	Plasmoids
Viral Agent	Bacteria	+	Biomass
Water-Cooled CPU	Water	+	Reactive Metals

3rd Level Refined	2nd Level Refined		2nd Level Refined		2nd Level Refined
Biotech Research Reports	Construction Blocks	+	Livestock	+	Nanites
Camera Drones	Rocket Fuel	+	Silicate Glass
Condensates	Coolant	+	Oxides
Cryoprotectant Solution	Fertilizer	+	Synthetic Oil	+	Test Cultures
Data Chips	Microfiber Shielding	+	Supertensile Plastics
Gel-Matrix Biopaste	Biocells	+	Oxides	+	Superconductors
Guidance Systems	Transmitter	+	Water-Cooled CPU
Hazmat Detection Systems	Polytextiles	+	Transmitter	+	Viral Agent
Hermetic Membranes	Gen. Enhanced Livestock	+	Polyaramids
High-Tech Transmitters	Polyaramids	+	Transmitter
Industrial Explosives	Fertilizer	+	Polytextiles
Neocoms	Biocells	+	Silicate Glass
Nuclear Reactors	Enriched Uranium	+	Microfiber Shielding
Planetary Vehicles	Mechanical Parts	+	Miniature Electronics	+	Supertensile Plastics
Robotics	Consumer Electronics	+	Mechanical Parts
Smarfab Units	Construction Blocks	+	Miniature Electronics	+
Supercomputers	Consumer Electronics	+	Coolant	+	Water-Cooled CPU
Synthetic Synapses	Supertensile Plastics	+	Test Cultures
Transcranial Microcontrollers	Biocells	+	Nanites
Ukomi Super Conductors	Superconductors	+	Synthetic Oil
Vaccines	Livestock	+	Viral Agent

4th Level Refined	3rd Level Refined		3rd Level Refined		Misc. Refined (Level)
Broadcast Node	Data Chips	+	High-Tech Transmitters	+	Neocoms (3rd)
Integrity Response Drones	Gel-Matrix Biopaste	+	Hazmat Detection Systems	+	Planetary Vehicles (3rd)
Nano-Factory	Industrial Explosives	+	Ukomi Super Conductors	+	Reactive Metals (1st)
Organic Mortar Applicators	Condensates	+	Robotics	+	Bacteria (1st)
Recursive Computing Module	Guidance Systems	+	Synthetic Synapses	+	Transcranial Microcontrollers (3rd)
Self-Harmonizing Power Core	Camera Drones	+	Hermetic Membranes	+	Nuclear Reactors (3rd)
Sterile Conduits	Smartfab Units	+	Vaccines	+	Water (1st)
Wetware Mainframe	Biotech Research Reports	+	Cryoprotectant Solution	+	Supercomputers (3rd)

New Reference Category

I am adding a new category to this blog site named Reference. It is intended to be the quick link to information I find useful to reference while actually playing Eve. So while playing, a visitor can click the Reference category, and have all the most useful blog posts up for reference while playing. Feel free to provide feedback about what other topics or prior blog posts you think would be useful to include in this category.

Note that if you click the category from the side bar, it will not show all posts.  You will have to click the link to view older posts to see the older ones.  I'm trying to figure out how to show them all on the same page without that additional hassle, but so far I haven't figured out how.  If necessary, I will just go back and repost all the old posts so that they appear in the same date range and will all appear together, but I'm going to spend a little more time researching first to see if I can fix it without such a hackish workaround.

 Also, don't forget that I also have a Tools and Programming category that can also be useful, though those aren't necessarily needed on a regular basis, so most of them will most likely not also be included under Reference.

NPC Damage Types and Weaknesses

This year, I'm going to try to post more immediately useful Eve information.  Even though it's available many other places on the web already, I want my own set of pages hosted here that provide all the reference information of interest.  Today that means a table of NPC damage types and weaknesses.

NPC	Damage Type				Weakness Type
Faction	EM	Ki	Th	Ex	EM	Ki	Th	Ex	Ship Alias
Amarr	X		X				O
Angel	X	X	X	X				O	Gist
Blood Raider	X		X		O				Corp
Caldari		X	X			O
Gallente		X	X		O
Gurista		X	X			O
Khanid	X		X				O
Minmatar	X	X	X	X			O
Mordu		X	X			O
Rogue Drone	X	X	X	X	O
Sansha	X		X		O
Serpentis		X	X			O			Core